# Ghost Search — MCP-native, Tor-routed search layer for RAG (seeking research collaborators) **URL:** \ **Category:** Research\ **Created:** [August 21, 2026, 9:28pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104 "2026-08-21T21:28:13Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 21, 2026, 9:28pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/1 "2026-08-21T21:28:13Z")
I’m a researcher who got tired of paying 60% of my LLM budget for search access. So I built Ghost Search. I’m opening it as a community research project and looking for collaborators. ## The Problem Search is the bottleneck for RAG, not the LLM. A 7B model with good retrieval beats a 400B model with bad retrieval. But building a search layer that handles IP blocks, captcha walls, and cookie tracking requires infrastructure most researchers don’t have. I’m not the first to address this. OnionSearch (megadose, 1.7k stars), darkdump (josh0xA), darker (saadejazz), and Robin (apurvsinghgautam) all scrape onion search engines. LibreX (hnhx, 841 stars) is a privacy-respecting meta-search with Tor and I2P support. Dark Watchdog (IEEE 2025) uses RAG for dark web forum monitoring with a fine-tuned BERT classifier. IntelShed (posted here on HF Forums) combines hybrid BM25 + pgvector + cross-encoder reranking for OSINT. So the building blocks exist. What’s missing is the combination that makes it directly usable in an LLM agent workflow without paid APIs, without persistence, and without a backend. ## What I Built Ghost Search is a Tor-routed, privacy-preserving meta-search aggregator with 14 onion search engines, BM25 ranking, and an MCP server interface. No API keys for the search layer, no per-query fees, no commercial dependency. What makes it different from the projects above: - **MCP server, not just a CLI.** 7 tools (ghost\_search, ghost\_health, ghost\_engines, ghost\_reset\_engine, ghost\_classify, ghost\_stix\_export, ghost\_load\_blacklist), stdio transport, works with any MCP-compatible client. OnionSearch and darkdump are CLI tools. Robin has a Streamlit UI. None expose an MCP interface. If you use Claude Desktop, VS Code, or any MCP client, ghost\_search drops into your workflow directly. - **RAM-only, no persistence.** No PostgreSQL, no Redis, no Elasticsearch. Search state lives in memory and is purged on screen-off, process exit, or panic wipe. darkscraper, voidaccess, and m-lally/dark-web-search all persist to databases. Ghost doesn’t. This matters for the privacy use case — there’s nothing to seize, nothing to subpoena. - **Android app in F-Droid review.** No backend, no accounts, no telemetry. Links open in Tor Browser via intent. I’m not aware of another onion search aggregator that targets F-Droid with a RAM-only architecture. - **BM25 ranking with engine agreement boost.** Okapi BM25 (k1=1.2, b=0.75) replaces heuristic frequency sorting. m-lally/dark-web-search uses RediSearch BM25 but on a crawled index. Ghost ranks live meta-search results — no crawl, no index, no storage. - **STIX 2.1 export.** OASIS-standard Observable bundles (url, ipv4-addr, cryptocurrency-wallet, vulnerability, email-addr) with relationships. voidaccess also supports STIX 2.1, but their pipeline is heavier (13-stage, PostgreSQL, relationship graphs). Ghost’s export is stateless — one tool call, one bundle, nothing stored. - **LLM integration with Ollama fallback.** NVIDIA NIM (Nemotron-3-Super- 120B) primary, Ollama (qwen3:8b) local fallback with 60s cooldown on primary failure. Robin supports OpenAI/Claude/Gemini/Ollama but requires you to send queries through a third-party API. Ghost’s fallback means you can run the whole thing on a laptop with Ollama and never touch a paid API. - **Entity extraction is ReDoS-safe.** BTC, XMR, PGP, email, .onion, CVE, IPv4 — linear-time regex, bounded quantifiers. darkscraper extracts similar entities but stores them in PostgreSQL. Ghost extracts for display only, nothing persisted. - **All filters OFF by default.** Ahmia abuse blacklist, snippet liability filter, STIX export — every feature is opt-in. Results are marked, never removed. This was important for F-Droid acceptance. 180 tests, TypeScript, tsc clean. Repo: [https://codeberg.org/sookoothaii/ghost-search-mcp](https://codeberg.org/sookoothaii/ghost-search-mcp) ## What Already Exists vs. What’s Missing There are public dark web datasets — Dizzy (32,555 onion domains with category labels, arXiv 2209.07202), DUTA (26-class illegal activity classification, EACL 2017), CoDA (10,000 documents for linguistic analysis, NAACL 2022). These are classification datasets. They tell you what a domain IS. What doesn’t exist is a search ranking benchmark: “given query X, which results are most relevant, and does BM25 rank them correctly?” That’s the gap I want to fill. I can collect raw results through Tor. I can’t annotate them alone and call it rigorous. ## Who This Is For I have a workstation — i9, RTX 3080 Ti, 16GB VRAM. I can run local models, fine-tune, experiment. I use that hardware to build the infrastructure. But the infrastructure isn’t for me. It’s for the researcher in a developing country with a laptop and no GPU. It’s for the student who can’t afford GPT-5 search API calls. It’s for the journalist who needs anonymous search without exposing their identity to a commercial API provider. Free, anonymous, secure, no paid dependency. That’s the design goal. I build with my hardware so others can use it without hardware. ## Where I Need Help I’m one person. I can build infrastructure. I can’t do all of this: 1. **Search ranking benchmark dataset** — I’m building an annotated dataset of onion search results: query, results, relevance scores. I need help with annotation guidelines and the actual annotation. Multiple annotators per result, agreement metrics, published rubric. I can collect the data. I can’t annotate it alone. 2. **Ranking evaluation against alternatives** — BM25 is my baseline. Cross-encoder re-ranking is a well-established next step (BEIR benchmark, MS MARCO, sentence-transformers CrossEncoderRerankingEvaluator). The game-retrieval benchmark by kimeyu showed BM25 beats dense retrieval in-domain, and reranking value is domain-dependent. Onion search might be another domain where BM25 is hard to beat. I need someone who has done retrieval evaluation to design the experiment properly. 3. **Query refinement with a fine-tuned model** — I’m currently using Nemotron via API for query expansion. The INTERS dataset (ACL 2024) showed instruction tuning on 20 IR tasks significantly boosts LLM performance on query understanding. SAIL (EMNLP 2023) fine-tuned LLaMA-7B with search-augmented training and outperformed ChatGPT on fact-checking. FOLLOWIR (NAACL 2025) fine-tuned a 7B model to follow complex IR instructions. I want to know: can a fine-tuned 7B model replace my Nemotron dependency for query refinement? I have the pipeline and the hardware to train. I need someone who has done instruction tuning to guide the methodology. 4. **MCP integration testing** — If you use Claude Desktop, VS Code, or any MCP client, I want to know: does ghost\_search work in your RAG workflow? What’s missing? I’ve tested it locally but I haven’t seen it in someone else’s pipeline. 5. **Privacy audit** — I believe the architecture is anonymity-preserving (Tor-only, RAM-only, no telemetry, no accounts). I’m a developer, not a security researcher. I’d welcome a review from someone with OPSEC/Tor expertise. ## How to Participate - Star/watch the repo: [https://codeberg.org/sookoothaii/ghost-search-mcp](https://codeberg.org/sookoothaii/ghost-search-mcp) - Reply below — tell me what you’d use this for. That helps me prioritize what to build next. - If you want to contribute: the engine layer is modular TypeScript, each engine is ~20 lines, adding a new one is straightforward - Dataset annotation: DM me if interested, I’ll coordinate - If you have OPSEC/Tor expertise and want to audit the architecture, I’d genuinely value that This is a FOSS project. MIT licensed. No commercial dependency, no surveillance, no data sale. Privacy is the product, not the cost. I’m one person. I can build the infrastructure. I can’t build the community alone. If this is useful to you, tell me. If it’s not useful yet, tell me what’s missing. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 22, 2026, 11:42pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/2 "2026-08-22T23:42:26Z")
For now, from the perspective of making this easier to move forward as a research project, I think it’s probably something like this: * * * Yes — I think there is a real research project here, and I would probably separate it into **two measurement layers before investing heavily in either annotation or a neural reranker** : 1. **live acquisition** — what the onion engines actually return, and whether they were reachable / challenged / parseable at that moment; 2. **frozen retrieval/ranking** — given exactly the same captured candidate set, how BM25, fusion, agreement, heuristics, cross-encoders, etc. reorder it. That distinction seems unusually important here because a Tor metasearch system is not just a ranker. It is closer to an **uncooperative federated-search / result-merging system** : several independent search engines produce heterogeneous result lists, then a broker has to normalize, deduplicate and merge them. That is very close to the problem setting of the [TREC Federated Web Search track](https://trec.nist.gov/pubs/trec22/papers/FEDERATED.OVERVIEW.pdf), including the annoying real-world part where the central system may have only URL/title/snippet rather than comparable upstream scores. There is even later work specifically on [snippet-based result merging in uncooperative federated search](https://journals.sagepub.com/doi/10.1177/01655515221144864). So the default route I would use is roughly: ```plaintext A. Runtime / acquisition sanity ↓ B. Measure live per-engine acquisition ↓ C. Freeze the raw SERPs / candidate pool ↓ D. Build pooled relevance judgments ↓ E. Replay the same pool offline: raw ranks / RRF / BM25 / agreement / quality heuristic / cross-encoder ↓ F. Then compare query refinement: raw query / Nemotron / local 7B ``` with **MCP integration/failure semantics** and **privacy/OPSEC** as parallel tracks rather than variables silently mixed into the ranking score. If I were doing only _one_ cheap check first, though, I would check the boring part: **are the deployed search paths actually producing non-empty titles and snippets from the engines before BM25 sees them?** I ran a small CPU-only/offline sanity pass against the code version I was looking at (`77e68649b8bf4fefda75a1b6f3c6a33fcdc11da2`), pinning Bun 1.3.14 and using synthetic HTML rather than live onion traffic. In that environment, global `DOMParser` was unavailable and `linkedom` did not resolve; the selector parser therefore fell back to the regex path, which produced the URL as the title and an empty snippet. Bun’s documented server-side Web APIs/globals are consistent with `DOMParser` not being available by default. I would **not** generalize that to your actual deployed environment without checking it there, but it makes this one-line sanity test very high-value: ```plaintext for each engine: status raw_result_count title_present? snippet_present? ``` If those fields look healthy in your actual runtime, great — move straight on. If not, fixing candidate acquisition first will make every later BM25/reranker comparison much easier to interpret. > **Why I think the federated-search framing may be useful** > > The connection is more than terminology. > > The classic federated-search problem is normally split into: > > - **resource selection** — which remote search engines/resources are useful for this query? > - **result merging** — how do we combine results from independently ranked sources whose internal scores/statistics may not be comparable? > > The [TREC FedWeb overview](https://trec.nist.gov/pubs/trec22/papers/FEDERATED.OVERVIEW.pdf) used results from 157 real search engines and explicitly separated those two tasks. For result merging, participants were given search result pages containing titles, snippets and hyperlinks and had to construct a unified ranking. > > That is remarkably close to the shape here. > > I would not immediately add query-dependent engine selection to Ghost Search — querying all engines is a perfectly useful way to collect baseline data first. But I _would_ preserve the engine identity and raw rank for every candidate. Then later you can measure: > > ```plaintext > engine availability > engine latency > raw yield > unique yield > relevant yield > pairwise overlap > > ``` > > and decide empirically whether source selection is worth adding. > > This also gives a useful interpretation of “engine agreement.” > > The intuition that a document returned by several systems may deserve more weight has a long history in **data fusion / rank fusion**. A particularly cheap baseline is [Reciprocal Rank Fusion (RRF)](https://research.google/pubs/reciprocal-rank-fusion-outperforms-condorcet-and-individual-rank-learning-methods/), and RRF was also among the submitted TREC FedWeb result-merging runs. > > So I would not replace the agreement idea. I would turn it into an ablation: > > ```plaintext > raw upstream ranking > RRF > BM25 > BM25 + raw engine agreement > possibly BM25 + agreement adjusted for engine overlap > > ``` > > The last control matters because “three engines returned this URL” is not necessarily three independent votes. Onion services have mirrors/duplicates, and search engines may themselves have correlated coverage. A simple pairwise URL-overlap/Jaccard matrix would already tell you a lot before inventing a sophisticated correction. > > In other words, agreement looks like an interesting research variable, not something that needs to be removed. > **How I would build the ranking benchmark** > > This is probably the part of your post where I think the project can become most reusable to other researchers. > > ### 1. Preserve the raw SERPs before normalization > > Before canonicalization/dedup/reranking, I would retain something approximately like: > > ```plaintext > query_id > query_text > timestamp > engine > raw_rank > raw_url > title > snippet > acquisition_status > Ghost version/config > > ``` > > Then derive: > > ```plaintext > canonical_url > duplicate_class > engine_count > > ``` > > later. > > That gives you the option of changing dedup/fusion/ranking without losing the original observations. > > It also matters because this is a temporally unstable collection. A qrel saying “document X was relevant to query Q” is much more useful if it really means: > > > this result surrogate was observed from engine E at time T, under configuration C. > > That lets a future replication distinguish **ranking drift** from **web/index drift**. > > ### 2. Define what the annotator is judging > > I think there are at least three subtly different targets here: > > 1. **SERP-surrogate relevance** > From URL/title/snippet, does this look relevant to the query? > > 2. **underlying-page topical relevance** > After fetching the document, is the actual page relevant? > > 3. **downstream RAG utility** > Did the retrieved content help answer the user’s task? > > Those do not have to become three benchmarks immediately. > > For a first open benchmark, I would probably make **SERP-surrogate relevance** the explicit target, because that is exactly what the central Ghost ranking layer often has available, and it avoids making full-page collection a prerequisite. > > That also makes the benchmark safer and easier to redistribute: harmless research queries plus result surrogates can establish a lot before anyone needs a broad archive of onion content. > > Later you could add a smaller page-level subset where it is appropriate. > > ### 3. Pool across diverse runs before judging > > The main methodological trap I would avoid is: > > ```plaintext > Ghost BM25 top-k > ↓ > human judges those documents > ↓ > same qrels used to prove BM25 beats alternatives > > ``` > > That can favor the system that created the judgment pool. > > TREC’s normal solution is **pooling** : take high-ranked results from several sufficiently different runs, union them, then judge that pool. NIST has also documented how shallow/incomplete pools can bias evaluation ([Buckley et al., “Bias and the Limits of Pooling for Large Collections”](https://www.nist.gov/publications/bias-and-limits-pooling-large-collections)). > > A cheap first Ghost pool could be: > > ```plaintext > union( > each engine's raw top-k, > simple fusion top-k, > BM25 top-k, > one substantially different reranker top-k > ) > > ``` > > You do not need every imaginable model in the first pool. The important thing is that the pool is not defined solely by the same retrieval function you are evaluating. > > TREC FedWeb is again useful here because its public evaluation resources include not only relevance judgments but also [duplicate-document classes](https://trec.nist.gov/data/federated2013.html). > > ### 4. Graded relevance seems worth it > > Something simple such as: > > ```plaintext > 0 = not relevant > 1 = marginal / weakly useful > 2 = relevant > 3 = highly relevant > > ``` > > would support nDCG while still being reasonably annotatable. > > I would keep these separate: > > ```plaintext > relevance > availability > duplicate/mirror status > challenge/error status > > ``` > > A dead page is not automatically irrelevant; a CAPTCHA is not a relevance judgment; two mirrors are not two separate relevance judgments. > > That separation becomes especially useful later. > > ### 5. Multiple annotators > > Your idea of multiple annotators + agreement metrics makes sense. > > I would publish: > > - the rubric; > - the anonymized per-annotator judgments if possible; > - the aggregated judgment; > - agreement statistics; > - adjudication policy, if disagreements are adjudicated. > > That makes the dataset much easier to reinterpret later if someone disagrees with the relevance boundary. > > ### 6. Keep a live track and a frozen track > > This is the part I would emphasize most. > > **Live acquisition track:** > > ```plaintext > query × engine > success? > latency > challenge? > HTTP failure? > parse failure? > raw result count > unique contribution > > ``` > > **Frozen ranking track:** > > Take one timestamped captured candidate set and replay: > > ```plaintext > raw engine order > RRF > BM25 > BM25 + agreement > quality heuristic > cross-encoder > LLM reranker > > ``` > > Now when model B beats model A, you know the candidate set did not move underneath the experiment. > > That gives you two useful results rather than one ambiguous result: > > > How good/reliable is live onion candidate acquisition? > > and > > > Given the same candidates, how good is the Ghost ranking layer? > **A few implementation boundaries that seem worth separating before interpreting ranking numbers** > > These are not reasons to change the overall direction; mostly they are reasons I would establish the measurement contracts before collecting a large qrel set. > > ### Parser/runtime > > As mentioned above, in my pinned CPU run: > > ```plaintext > typeof DOMParser -> undefined > linkedom -> unresolved > selector fixture -> URL title + empty snippet fallback > > ``` > > So I would first verify this in the actual environment. > > If the real deployment does have a DOM implementation supplied some other way, this concern disappears immediately. > > If it does not, it is important because BM25 over: > > ```plaintext > title + snippet > > ``` > > is then not the intended title/snippet experiment. > > ### MCP and Dashboard currently appear to have different acquisition paths > > In the code version I checked, the Dashboard invokes engine-specific `prepareSearchUrl()` hooks while the MCP search path does not. > > This matters particularly for engines that need a preflight/form step. > > Ahmia is a concrete example: its current public code generates a rolling six-character token and rotating field name, and redirects requests with an invalid/missing token. You can see that logic directly in the current [Ahmia search view](https://github.com/ahmia/ahmia-site/blob/master/ahmia/views.py). > > Ghost already has an Ahmia preparation hook intended to obtain that hidden input, which looks like the right basic idea; the useful question is simply whether the **same acquisition core** reaches both Dashboard and MCP. > > So I would either: > > ```plaintext > unify: > Dashboard ─┐ > ├─ search core → engines > MCP ───────┘ > > ``` > > or, if keeping separate paths is intentional, record the path/config in benchmark metadata. > > That turns “UI differences” into an explicit experimental variable rather than accidental noise. > > ### Preserve duplicate provenance before dedup > > In this snapshot, duplicate URL handling keeps the first result, while the BM25 input is later supplied with: > > ```plaintext > engineCount = 1 > trustScore = 1 > > ``` > > for each document. > > I reproduced that behavior with a tiny duplicate fixture. > > So the interesting interpretation is not “agreement is a bad idea”; it is almost the opposite: > > > preserve the provenance first, and agreement becomes an actual variable you can test. > > For example: > > ```plaintext > canonical result > url > observed_by: > - engine A, rank 3 > - engine D, rank 1 > - engine H, rank 7 > > ``` > > gives you far more options than collapsing that to the first engine immediately. > > ### Treat the quality filter as a ranking arm in the evaluation > > I also tested the current quality filter with a controlled two-result fixture. A result that BM25 placed first could become second after the quality filter, because the quality stage re-sorts by its own quality score. > > That is not necessarily wrong — it may be useful. > > But experimentally I would label it as something like: > > ```plaintext > BM25 > quality heuristic > BM25 + preserved agreement > cross-encoder > > ``` > > rather than assuming that “quality filtering enabled” is still basically the same BM25 ranking. > > Then the ablation tells you what each stage actually buys you. > > ### Distinguish challenge/parse failure from a genuine empty result > > The code already contains challenge/CAPTCHA detection logic, which is useful. > > The measurement question is whether an HTML challenge that returns HTTP 200 can reach the result parser, yield zero parsed results, and be counted as an engine success. > > For an agent, these are very different observations: > > ```plaintext > valid_empty > challenge > timeout > HTTP_error > parse_empty > partial_success > > ``` > > For an IR benchmark they are also different: none except `valid_empty` really says “the engine searched successfully and found nothing.” > > That distinction is cheap to add and can save a lot of debugging later. > **On the Nemotron → local 7B question** > > I think this is testable, but I would actually delay the fine-tuning step by one experiment. > > First measure: > > ```plaintext > raw query > vs > Nemotron-refined query > > ``` > > on the **same frozen retrieval evaluation**. > > Then: > > ```plaintext > Does Nemotron produce a reproducible downstream retrieval lift? > > no > └─> there may be no useful effect to distill yet > > yes > └─> add the local 7B and test whether it preserves that lift > > ``` > > That answers a more fundamental question before spending training effort: > > > Is query refinement helping Ghost Search in this domain at all? > > Only after that do you really need: > > > How small/local can the refiner become? > > There is good precedent for smaller instruction-tuned models acquiring useful IR abilities, but I would treat it as evidence that the experiment is plausible, not as evidence that a 7B model will automatically replace Nemotron here. > > For example: > > - [INTERS](https://aclanthology.org/2024.acl-long.154/) instruction-tunes models across query understanding, document understanding and query-document tasks and reports substantial improvements on IR tasks. > - [FollowIR](https://aclanthology.org/2025.naacl-long.597/) builds an instruction-following retrieval benchmark and fine-tunes a 7B retrieval model, improving its ability to follow detailed retrieval instructions. > > But neither establishes: > > ```plaintext > 7B query rewriter == Nemotron query rewriter > > ``` > > for onion search. > > And newer reproducibility work is a useful caution: [a 2026 controlled study of LLM query reformulation](https://arxiv.org/abs/2604.27421) found that gains depend strongly on whether the downstream retriever is lexical, learned-sparse or dense, and that larger LLMs are not uniformly better. > > So I would make the end-to-end retrieval score the primary outcome, not “the rewrite looks smarter.” > > A minimal matrix could be: > > | Query path | Same candidate-generation configuration | Retrieval metric | Latency/cost | > | --- | --- | --- | --- | > | Raw | yes | nDCG / Recall / MRR | baseline | > | Nemotron | yes | same | measured | > | Local 7B | yes | same | measured | > > If Nemotron has a clear lift, you also get training data almost naturally: > > ```plaintext > raw query → effective refinement > > ``` > > plus retrieval outcome as an external signal. > > That seems like a much stronger basis for the local model than training it only to imitate fluent reformulations. > **For MCP / RAG integration, I would make partial failure visible** > > The practical Agent-side issue I would prioritize is **failure semantics**. > > A metasearch tool can legitimately return useful results even if several engines fail, so I would not turn every engine error into a failure of the entire MCP call. > > Instead, something conceptually like: > > ```json > { > "results": [...], > "raw_candidate_count": 42, > "unique_candidate_count": 27, > "engine_status": { > "engine_a": "ok", > "engine_b": "challenge", > "engine_c": "timeout", > "engine_d": "valid_empty" > }, > "partial_failures": [...], > "elapsed_ms": 1832 > } > > ``` > > would let an Agent distinguish: > > > “there appears to be no result” > > from: > > > “most of my search infrastructure failed.” > > That distinction matters a lot in agentic systems because otherwise the model may treat an infrastructure failure as evidence about the world. > > MCP already has useful primitives for this direction. The [Tools specification](https://modelcontextprotocol.io/specification/2025-11-25/server/tools) supports `structuredContent`, optional `outputSchema`, and `isError` for tool-execution failures. It also explicitly distinguishes protocol errors from actionable tool errors. > > So, for example: > > ```plaintext > 14 engines > ├─ 9 return usable results > ├─ 2 challenge > ├─ 2 timeout > └─ 1 valid empty > > ``` > > could still be an overall successful search result with structured partial-failure metadata. > > Whereas: > > ```plaintext > Tor unavailable > all engines unreachable > invalid request > > ``` > > might reasonably become an actual tool execution error. > > That would also make real-world MCP testing much more informative: instead of only asking “did Claude/another client invoke the tool?”, you can inspect whether the Agent makes sensible decisions under degraded retrieval. > > A few particularly useful integration fixtures would be: > > ```plaintext > 1. normal results > 2. genuine empty results > 3. half the engines time out > 4. challenge page returned as HTTP 200 > 5. malformed result HTML > 6. result snippet containing instruction-like text > > ``` > > The last one is not specific to Tor; it is simply because search results are open-world, untrusted content. MCP’s own security guidance recommends output sanitization and validation of tool results before passing them onward to an LLM. > > I would keep that as an integration test rather than trying to make the search engine itself decide what text is “safe.” > **For privacy / OPSEC, I would separate Tor transport from the optional LLM data path** > > I think this part benefits from being described mode-by-mode rather than with one global “anonymous” label. > > On the Tor side, some of the current design choices look directionally sensible: > > - destination hostnames are passed through SOCKS rather than obviously pre-resolved; > - per-request SOCKS credentials can be used by Tor for stream isolation; > - the supplied Tor configuration includes `SafeSocks 1`. > > There is also a very cheap positive test available directly from Tor. > > The Tor Project documents [`TestSocks 1`](https://support.torproject.org/little-t-tor/troubleshooting/check-for-leaks/) specifically for checking whether a SOCKS-using application leaks DNS: Tor logs a notice for safe SOCKS requests and a warning for unsafe ones. `SafeSocks 1` can reject unsafe connections. > > So an OPSEC smoke test could be as simple as: > > ```plaintext > TestSocks 1 > SafeSocks 1 > > run several harmless Ghost searches > inspect Tor log > > ``` > > That gives you stronger evidence than just inferring behavior from the socket code. > > I would then treat LLM refinement/summarization as a separate trust boundary. > > In a mocked network test of the code version I checked, the MCP defaults could construct the LLM client without a NIM key because a local Ollama model is configured, but the client still attempted the remote NVIDIA endpoint first and included the raw query in that request body before falling back to local Ollama. > > That does **not** say anything about what NVIDIA subsequently stores or does with data; I would not infer that from the client code. > > It simply means the privacy model is clearer if these modes are explicit: > > ```plaintext > Mode A: > Tor retrieval only / local processing > > Mode B: > Tor retrieval + local LLM > > Mode C: > Tor retrieval + remote LLM refinement/synthesis > > ``` > > Then the documentation can say exactly which data crosses which boundary in each mode. > > For a project whose design goal includes privacy, that distinction strengthens the claim rather than weakening it. > > The same principle applies to observability: status codes, counts and timings are relatively easy to keep; raw query/URL/snippet logging should probably be an explicit choice rather than something introduced accidentally while making the system easier to debug. > **If I wanted the maximum information from the first small experiment** > > I would avoid starting with a giant annotation campaign. > > A small first pass could use perhaps 20–50 **benign research queries** , with an even smaller 1–3 query smoke test before that. > > ### Phase 0 — runtime sanity > > ```plaintext > DOM parser available? > per-engine title present? > per-engine snippet present? > challenge detector connected? > > ``` > > If this fails, stop there and fix acquisition. > > ### Phase 1 — live acquisition > > For every `query × engine`: > > ```plaintext > timestamp > status > latency > raw_count > title/snippet presence > challenge/error class > > ``` > > Then compute: > > ```plaintext > unique URLs per engine > pairwise engine overlap > unique contribution per engine > > ``` > > No relevance annotation is required yet. > > This alone tells you whether “14 engines” behaves more like: > > ```plaintext > 14 reasonably independent sources > > ``` > > or: > > ```plaintext > 5 productive sources + 4 correlated sources + > 3 frequently challenged sources + 2 mostly dead sources > > ``` > > which is useful research information in its own right. > > ### Phase 2 — freeze the candidates > > Save the raw SERPs. > > Now network state is no longer part of the ranking experiment. > > ### Phase 3 — cheap ranking baselines > > Before paying for cross-encoder or LLM evaluation: > > ```plaintext > raw engine ordering > round-robin / trivial merge > RRF > BM25 > BM25 + preserved engine agreement > quality heuristic > > ``` > > These are cheap enough that they make excellent controls. > > If a neural reranker cannot beat those reliably, that itself is a useful result. > > ### Phase 4 — human judgments > > Construct a diverse pool and annotate it. > > Then use metrics appropriate to the judgment scheme, e.g.: > > ```plaintext > nDCG@k > MRR > Recall@k > > ``` > > I would report acquisition/reliability metrics separately rather than trying to fold them into one magic score. > > ### Phase 5 — expensive models > > Only now: > > ```plaintext > cross-encoder > LLM reranker > raw vs Nemotron > raw vs Nemotron vs local 7B > > ``` > > This ordering has a nice property: every expensive step is justified by information from the cheaper previous step. > > A useful decision tree is: > > ```plaintext > candidate acquisition unstable? > yes → measure/fix acquisition first > no → freeze candidate pools > > ranking baselines meaningfully differ? > yes → build qrels / compare rankers > no → investigate candidate recall / query set first > > Nemotron gives measurable retrieval lift? > no → local-7B training can wait > yes → local 7B becomes a justified compression/replacement experiment > > ``` A couple of smaller observations I would probably keep secondary rather than letting them distract from the research design: - If engine-specific preparation is moved into the MCP path, keep stdio protocol output clean; normal diagnostic logging should not share stdout with the MCP wire. - Session/cookie-dependent challenge flows deserve a small mock test because a stateless HTTP transport and a browser-like challenge flow have different contracts. - I would avoid calling the benchmark “the first” until the literature search is completely nailed down. [Morato et al.](https://doi.org/10.1108/IDD-07-2024-0096) already evaluated retrieval/ranking behavior across Tor search engines, even though I have not found a reusable public onion query–qrels collection equivalent to what you are proposing. - If later you optimize source selection rather than querying all engines, that can be treated as a new experimental layer rather than silently changing the benchmark. So, condensed down to the part I think has the highest information gain: **I would keep the overall project direction, but make the boundaries measurable before scaling it.** First verify the result-surrogate acquisition path, preserve raw per-engine provenance, distinguish live retrieval failures from true empty results, and freeze candidate pools before comparing ranking methods. Then the benchmark can tell you separately whether Ghost improves **coverage** , **fusion/ranking** , and **query refinement** , instead of one stage accidentally taking credit or blame for another. Once those boundaries are in place, the things you are already interested in — engine agreement, BM25 vs cross-encoder, Nemotron vs a local 7B, and MCP/RAG integration — become much cleaner experiments rather than competing moving parts. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 24, 2026, 1:40pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/3 "2026-08-24T13:40:57Z")
Thanks for the detailed review — this is exactly the kind of methodology guidance I needed. I’ve been working through your suggestions and have concrete progress on several points, including new results from the last two days. **Phase 0 (runtime sanity) — fixed and verified** The DOMParser/linkedom bug you identified was real and is now fixed. linkedom@0.18.13 is in package.json, parseDocument() works in Bun’s server-side runtime, and selector-based parsing produces non-empty titles and snippets. I verified this with a live Docker deployment: 14 engines queried, 13/14 returned non-empty results with real snippets (only tor66 returned error\_404). I also ported 180 tests from the Android app’s test suite (blacklist, camouflage, dedup, entities, query, ranking, STIX) and they all pass in the Bun runtime. The dedup.ts module was extracted with 6 pure functions that have no Android-specific dependencies. **Engine health and partial-failure semantics — implemented** Your suggestion to distinguish valid\_empty from challenge/timeout/error is now live in the API response. Every search returns an engineHealth dict with per-engine status, results count, and ms latency: ```auto ``` ```auto engineHealth: { ``` ahmia: {status: “ok”, results: 2297, ms: 7681}, tor66: {status: “error\_404”, results: 0, ms: 500}, torgle: {status: “ok”, results: 0, ms: 6025}, // valid\_empty … } This flows through to the IntelShed UI as a color-coded engine health panel. Partial failures render results immediately with the health panel below — no error modal for 9/14 engines succeeding. I also added raw\_candidate\_count and unique\_candidate\_count to the response, so an agent can distinguish “2537 raw candidates, 50 unique after dedup” from “0 results because infrastructure failed.” **LLM mode transparency — implemented** The /api/status endpoint now reports which LLM boundary is active: ```auto ``` ```auto llm_mode: "remote_nim" | "local_ollama" | "none" ``` llm\_model: “nvidia/nemotron-3-super-120b-a12b” | “qwen3:8b” llm\_boundary: “query\_text\_sent\_to\_nvidia” | “local\_only” This is informational, not a control — the operator changes it via env vars, not through the UI. But it makes the privacy boundary explicit, which is exactly what you suggested with the Mode A/B/C separation. **MCP vs Dashboard acquisition path — partially addressed** You were right that the Dashboard and MCP paths had different acquisition logic. The Dashboard’s handleSearch() calls engine.prepareSearchUrl() (needed for Ahmia’s anti-bot token), while the MCP path didn’t. This is now unified — both paths go through the same search core. The IntelShed integration uses the Dashboard’s HTTP REST API (POST /api/search), which has the complete acquisition path including prepareSearchUrl. The remaining question is whether the MCP stdio path should also be updated for standalone use. Currently the IntelShed integration doesn’t use MCP stdio at all — it uses HTTP REST. But for Claude Desktop / VS Code users, the MCP path should have the same acquisition quality. I’ll address this next. **On the federated search framing — agreed, and it shaped the architecture** Your TREC FedWeb analogy was directly useful. Ghost is now integrated into IntelShed as an “uncooperative federated search broker” — IntelShed calls Ghost via HTTP, Ghost handles all 14 engines, and IntelShed consumes the merged results for entity extraction and FtM matching. The intelligence layer (entity extraction, FtM graph, content retrieval) stays in Python; the search layer (acquisition, parsing, BM25, dedup) stays in TypeScript. One parser implementation, one codebase to maintain. The engine\_health dict is exactly the “live acquisition track” you described — it’s now a permanent part of the API response, not just a benchmark artifact. **SERP freezing and ranking replay — completed (new since last post)** Following your Phase 0-5 ordering, I’ve now completed Phase 2 (SERP freezing) and built a ranking replay tool: - **Frozen corpus** : 24 queries × 13 engines, 28,280 results collected, 99.4% snippet coverage. Queries span 4 categories (threat intelligence, security research, investigative journalism, infrastructure entities). The corpus is gitignored local research data — reproducible offline, no live network needed for ranking experiments. - **Replay tool** (`replay-ranking.ts`): Loads frozen SERPs and runs them through 3 ranking strategies without live Tor: **Key findings from the frozen corpus:** - **Dedup ratio** : The highest-volume engine (Ahmia, 24,544 raw results) has a 0.982 unique-URL ratio — its high yield is genuine diversity, not mirror duplication. The remaining 12 engines collectively produce 3,736 raw → 3,086 unique (0.826). Cross-engine overlap between Ahmia and the rest is 0-1.6% on most queries. - **Ranking strategy matters** : Mean Top-10 Jaccard overlap between strategies: engine-count∩BM25=0.463, engine-count∩RRF=0.390, BM25∩RRF=0.436. On one query (“zero-day”), engine-count and RRF produce completely disjoint Top-10 lists (Jaccard=0.0). - **Per-engine unique contribution** : Ahmia contributes 23,713 unique URLs (96.6% of its results are unique to Ahmia). OnionLand has 100% unique contribution but only 2 results/query average — small but perfectly non-overlapping. **Engine availability — updated (new since last post)** Two engines that showed 0 results in the frozen corpus (OSS, Torgle) were diagnosed as transient Tor circuit issues, not parser bugs — both now return results live (11 and 10 results respectively). One engine (Torch) had a stale endpoint address; updated to the classic Omega CGI interface, now returning 10 results in 5.1s with a 3.6M document index. Current live status: 12/13 engines operational, 370 merged results on a test query. I also added per-engine connect timeout overrides based on Tor Project recommendations (60-90s for onion services): the two slowest engines get individual timeouts (40s and 30s) while fast engines keep the 20s default — no performance impact on healthy engines, but the slow ones no longer get falsely marked as timed out. **On the ranking benchmark — agreed on ordering, starting with acquisition** I agree with your Phase 0-5 ordering. The SERP freezing (Phase 2) and replay tool are now done. The pairwise engine overlap computation (your Phase 1) is partially complete — the benchmark module has Jaccard similarity and uniqueContribution() functions, and I’ve run them on the frozen corpus. The next step is qrels annotation (Phase 3) — manual relevance scoring on the frozen Top-20 per query, which enables nDCG/MAP computation for the ranking comparison. **On query refinement — agreed, measuring Nemotron lift first** I’m not fine-tuning a 7B model yet. The current architecture has Nemotron as primary with Ollama qwen3:8b as fallback. Before investing in training, I’ll run your suggested matrix: raw query vs Nemotron-refined query on the same frozen candidate pool, measure nDCG/Recall/MRR. If there’s no measurable lift, the 7B distillation experiment can wait. **On operational configuration — Tor safety checks noted** The Tor configuration already includes SafeSocks 1. I’ll add the TestSocks 1 log inspection as an explicit smoke test. The per-search SOCKS5 credential isolation (RFC 1929) is already implemented and device-verified — different session IDs per search prove isolated circuits. **What’s next** 1. qrels annotation on frozen SERPs (manual, no network needed) 2. nDCG/MAP computation for the 3 ranking strategies 3. Nemotron query refinement lift measurement 4. MCP stdio path: unify acquisition with Dashboard/REST path The quality filter is now treated as a separate ranking arm in the evaluation, not silently mixed into BM25. And duplicate provenance is preserved before dedup — the engineHealth dict records which engines returned each URL, not just the first one. Thanks again for the structured feedback. The federated-search framing in particular changed how I think about the architecture — it’s not just a ranker, it’s a broker, and the measurement boundaries you suggested are now explicit in the API. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 25, 2026, 2:35pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/4 "2026-08-25T14:35:21Z")
I think this is a good direction: * * * For the remaining MCP question: **yes, I would bring the standalone stdio path to semantic parity with the REST path if Claude Desktop / VS Code / other MCP clients are still a supported use case.** I would not make “literally the same entry point” the requirement, though. The useful invariant is that the same query and the same engine responses produce the same normalized candidates, provenance, failure semantics, and ranking regardless of transport. For the benchmark, I think the highest-information next step is also still qrels — but I would insert one small calibration stage before annotating all 24 queries: 1. define the **information need** for each query, not just the query string; 2. define exactly what an assessor is judging; 3. build the judgment pool from several deliberately different ranking arms; 4. double-annotate 4–6 representative queries; 5. resolve rubric ambiguities and freeze the evaluation protocol; 6. then annotate the remaining topics. That is probably more useful right now than adding another neural reranker. The Top-10 Jaccard numbers already establish that the ranking strategies disagree substantially; qrels are what turn “these rankings are different” into “this difference is useful.” A compact default route might be: | Step | What it answers | Cost | | --- | --- | --- | | 4–6 query qrels calibration | Are we judging the same notion of relevance? | low | | pooled qrels for all 24 topics | Which ranker is actually better? | medium | | atomic BM25/RRF/agreement/trust ablations | Which component causes the lift? | low | | frozen + live Nemotron tests | Does refinement help ranking, acquisition, or both? | medium | | REST/stdio parity fixture | Do MCP users get the same search semantics? | low | | Tor ControlPort observation | Are circuit-isolation claims observed, not only configured? | optional | > **Qrels: I would define the benchmark before scaling the annotation** > > ### 1. Treat each item as a **topic** , not merely a query string > > This looks particularly important for short queries such as `zero-day`. > > Two annotators can apply exactly the same 0–3 rubric and still disagree completely if one interprets the query as: > > > current reporting about zero-day exploitation > > while another interprets it as: > > > technical information, research, tooling, markets, or discussion related to zero-day vulnerabilities. > > A useful precedent is the [TREC Federated Web Search track](https://trec.nist.gov/pubs/trec23/papers/overview-federated.pdf). FedWeb is unusually close to Ghost’s problem because it evaluated result merging across heterogeneous search resources, and its assessors were shown: > > - the query; > - a short description of the information need; > - a narrative explaining what should and should not count. > > So before labeling the full corpus, I would freeze something roughly like: > > ```plaintext > topic_id: q07 > query: zero-day > > description: > Information about zero-day software vulnerabilities. > > narrative: > Relevant results contain substantive information about zero-day > vulnerabilities, exploitation, disclosure, mitigation, research, > or closely related activity. > > Not relevant: > Generic security pages that merely contain the words "zero day", > unrelated products/brands, or pages for which the available result > surrogate gives no evidence of relevance. > > ``` > > It does not have to be verbose. Even two sentences can remove a surprising amount of annotation noise. > > For any topic whose real downstream objective is something narrower — for example entity extraction, FtM matching, finding a particular infrastructure artifact, etc. — I would keep that downstream objective separate from retrieval relevance. Otherwise one qrel starts trying to score both: > > > “Is this result relevant?” > > and > > > “Did Ghost successfully extract the entity I wanted?” > > Those are both useful measurements, but they are different measurements. > > * * * > > ### 2. Freeze the **judgment target** > > There are at least three reasonable benchmarks hiding here: > > **A. SERP-surrogate relevance** > > Judge only what Ghost’s broker actually has: > > ```plaintext > URL > title > snippet > source/provenance > > ``` > > **B. Underlying-page relevance** > > Open/fetch the result and judge the actual page. > > **C. Downstream utility** > > Ask whether the result helped an investigation, RAG answer, entity graph, etc. > > For the first Ghost benchmark, I would strongly prefer **A**. > > That matches the central broker’s actual information boundary, keeps annotation offline, and avoids turning the ranking benchmark into a Tor crawling / page-availability / content-extraction benchmark. > > It also makes the benchmark easier for other people to reproduce. > > If the frozen corpus stores a processed representation — e.g. HTML-stripped or length-limited snippets — I would name that explicitly in the dataset documentation. Something like **“frozen Ghost SERP surrogates”** is more precise than implying that the benchmark contains untouched raw engine responses. > > Then page-level evaluation can become a separate benchmark later if it becomes useful. > > * * * > > ### 3. Use a diverse pool, rather than Top-20 from only one ranking > > The usual TREC idea of [pooling](https://trec.nist.gov/howto.html) maps nicely here: judge the union of top results from several runs rather than treating one ranker’s candidates as the truth. > > For Ghost, the initial pool could simply be: > > ```plaintext > Top-k engine-count > ∪ Top-k BM25 > ∪ Top-k plain RRF > ∪ Top-k current Ghost RRF > ∪ Top-k quality-filter arm, if it retrieves genuinely different candidates > > ``` > > Then canonicalize/deduplicate the pool before presenting it to annotators. > > I would start with the 4–6 topic calibration first rather than committing immediately to a pool depth for all 24 topics. You will quickly see whether `k=10`, `20`, or something slightly larger produces a manageable pool. > > The useful property is **diversity of retrieval mechanisms** , not maximum pool size. > > A nice side effect is that a future cross-encoder does not immediately inherit BM25’s blind spots just because BM25 generated the qrels pool. > > * * * > > ### 4. Keep relevance and duplication separate > > I would not encode duplicates into the relevance label itself. > > Something like: > > ```plaintext > topic_id > candidate_id > canonical_url > duplicate_class_id > title > snippet > relevance_grade > judgment_status > annotator_id > > ``` > > is easier to reason about. > > For example: > > ```plaintext > relevance_grade: > 0 = irrelevant > 1 = marginal / weakly useful > 2 = relevant > 3 = highly relevant > > judgment_status: > judged > unjudgeable > > ``` > > `unjudgeable` is worth having. An empty or cryptic snippet should not silently become “irrelevant” if the assessor simply lacks enough evidence. > > The duplicate class can then be applied by the evaluation layer. > > FedWeb did something very similar conceptually: it maintained duplicate information separately and reported result-merging metrics where later occurrences of the same page received zero gain, as well as variants without the duplicate penalty. The [FedWeb 2014 data page](https://trec.nist.gov/data/federated2014.html) even publishes the duplicate-page sets separately from the relevance judgments. > > That gives Ghost a clean distinction between: > > ```plaintext > relevance: > "Is this candidate useful for this topic?" > > diversity: > "Is this candidate providing another independent result?" > > ``` > > That distinction will matter a lot for onion search. > > * * * > > ### 5. URL uniqueness and content uniqueness are worth reporting separately > > The Ahmia result is interesting: > > > 24,544 results, 0.982 unique-URL ratio, and very little URL overlap with the other engines. > > I would preserve that result, but label it specifically as **URL-level diversity**. > > A different URL does not necessarily imply different content. > > Because snippet coverage is already 99.4%, a cheap second measure does not require fetching any pages: > > ```plaintext > normalize(title + snippet) > -> exact hash > > optionally: > -> SimHash / MinHash / another cheap near-duplicate signature > > ``` > > Then report both: > > ```plaintext > URL-unique contribution > surrogate-content-unique contribution > > ``` > > Later, after qrels: > > ```plaintext > uniquely relevant contribution@k > > ``` > > That makes the engine-diversity result considerably more informative without changing the acquisition architecture. > > I would also normalize retrieval depth when comparing engines. > > Ahmia produced 24,544 of the 28,280 frozen results, so an absolute count such as: > > > “23,713 URLs unique to Ahmia” > > contains both: > > - diversity; > - much greater retrieval depth. > > A very cheap companion statistic would be: > > ```plaintext > unique@10 > unique@20 > unique@50 > > ``` > > per engine/query at equal depth. > > Then the interpretation becomes much cleaner: > > ```plaintext > absolute unique yield > -> what does this engine contribute in practice? > > unique@k > -> how non-redundant is it at comparable retrieval depth? > > relevant unique@k > -> how much uniquely useful material does it contribute? > > ``` > > I would not optimize engine selection until the third quantity exists. > > * * * > > ### 6. Pilot the rubric before annotating everything > > With only 24 topics, I would choose 4–6 that deliberately stress different cases: > > ```plaintext > high overlap between rankers > low overlap between rankers > very high-volume Ahmia case > low-yield / rare-source case > ambiguous short query > entity/infrastructure-style query > > ``` > > `zero-day`, where engine-count and RRF had Jaccard 0.0 in your replay, sounds like a particularly useful calibration topic. > > For those pilot topics: > > ```plaintext > two independent annotators > -> compare disagreements > -> discuss only after independent labeling > -> clarify the narrative/rubric > -> relabel if necessary > -> freeze rubric > > ``` > > You can calculate weighted agreement if useful, but I would treat the disagreement _examples_ as at least as important as the single agreement coefficient. They tell you where the benchmark definition itself is underspecified. > > If annotation manpower is limited, the pilot can be fully double-judged and the full collection can use whatever overlap strategy is affordable. The important thing is to state exactly what was independently judged and what was adjudicated. > > * * * > > ### 7. Metrics: make graded ranking the primary result > > For a 0–3 qrel, I would make something like: > > ```plaintext > nDCG@10 > nDCG@20 > > ``` > > the primary ranking metrics. > > MAP is still useful, but it needs an explicit binary conversion, for example: > > ```plaintext > grade >= 2 -> relevant > grade < 2 -> non-relevant > > ``` > > Otherwise two implementations can both report “MAP” while evaluating different things. > > I would also keep the per-topic values. > > Twenty-four topics are enough for a useful pilot benchmark, but they are not enough for a tiny mean difference to become automatically convincing. Classic TREC work on [topic-set size and retrieval experiment error](https://www.nist.gov/publications/effect-topic-set-size-retrieval-experiment-error) is a good reminder that retrieval effectiveness varies substantially by topic. > > So I would report: > > ```plaintext > mean nDCG@10 / nDCG@20 > median > per-query delta against baseline > > ``` > > and, if convenient: > > ```plaintext > paired bootstrap / randomization interval > > ``` > > A per-query delta table will probably be especially informative here because the existing Jaccard numbers already suggest that behavior is heterogeneous across topics. > > For future models that retrieve many documents outside the original judgment pool, also track something like: > > ```plaintext > judged@10 > judged@20 > > ``` > > so a score drop is not mistaken for “worse ranking” when it is actually “this new system retrieved mostly unjudged documents.” > > If the qrels eventually become substantially incomplete for new systems, [bpref / incomplete-judgment evaluation](https://www.nist.gov/publications/retrieval-evaluation-incomplete-information) is another tool available later. I would not complicate the first experiment with it unless it becomes necessary. > > * * * > > ### 8. Freeze evaluation before tuning > > One easy benchmark trap would be: > > ```plaintext > build qrels > -> tune BM25/agreement/trust/RRF weights on all 24 topics > -> report final performance on the same 24 > > ``` > > For an exploratory repo this is understandable, but the final number then measures some amount of tuning to the benchmark. > > A low-cost separation would be: > > ```plaintext > 4–6 calibration/development topics > remaining topics = evaluation > > ``` > > or simply: > > ```plaintext > freeze all parameters before looking at qrel-based effectiveness > > ``` > > for the first baseline comparison. > > The latter may actually be preferable initially because your current parameters already exist. > **Ranking replay: make the arms atomic** > > The current Jaccard result already answers one question well: > > > the ranking strategies are not equivalent. > > It does **not** yet tell us which component caused the change or which ranking is better. > > I would therefore make the replay arms as boring and atomic as possible. > > For example: > > | Arm | Purpose | > | --- | --- | > | engine-count | current/simple baseline | > | lexical BM25 | lexical-ranking baseline | > | BM25 + agreement | isolate agreement boost | > | plain RRF, `k=60` | literature baseline | > | RRF + trust | isolate trust weighting | > | quality-filter arm | measure filter independently | > > The original [Cormack, Clarke & Büttcher RRF paper](https://cormack.uwaterloo.ca/cormack/cormacksigir09-rrf.pdf) is useful here because plain RRF is deliberately simple: > > ```plaintext > score(d) = sum over input rankings of 1 / (k + rank(d)) > > ``` > > with `k=60` in their experiments. > > So if Ghost’s RRF includes trust weighting, source priors, snippet-based ordering, or another pre/post-processing step, I would keep those — they may be useful — but expose them as incremental arms: > > ```plaintext > plain RRF > -> + trust > -> + any Ghost-specific ordering heuristic > > ``` > > Then a result such as: > > > RRF + trust beats BM25 by X > > actually identifies something. > > The same logic applies to BM25: > > ```plaintext > BM25 lexical score > -> + engine agreement > -> + trust/source prior, if used > > ``` > > This also makes a future cross-encoder much easier to interpret. > > Instead of: > > ```plaintext > cross-encoder vs "Ghost ranking" > > ``` > > you get: > > ```plaintext > plain lexical baseline > fusion baseline > Ghost heuristics > neural reranker > > ``` > > which is much more reusable for a paper or dataset card. > > One small replay invariant I would add to the tests is: > > ```plaintext > after any operation that changes scores, > re-sort before Top-k evaluation > > ``` > > and use one canonical URL identity function everywhere that computes: > > ```plaintext > dedup ratio > engine overlap > unique contribution > ranking candidate identity > > ``` > > Those two invariants are cheap and prevent benchmark bookkeeping from becoming part of the ranking result. > **Nemotron: the frozen experiment and the live experiment answer different questions** > > I think delaying 7B fine-tuning until Nemotron demonstrates measurable value is exactly the right ordering. > > I would split the lift experiment into two independent tests. > > ### A. Frozen-candidate test > > ```plaintext > same frozen candidates > raw query > vs > Nemotron-refined query > > ``` > > This answers: > > > Does the rewritten query improve scoring/ranking among candidates that have already been discovered? > > That is a clean ranking experiment. > > ### B. Live-acquisition test > > ```plaintext > same information need > raw query -> engines -> candidate set A > > refined query -> engines -> candidate set B > > ``` > > This answers: > > > Does query refinement cause the underlying search engines to discover better candidates? > > For a metasearch broker, this may be the more important effect. > > A rewrite could have almost no effect on BM25 over a fixed pool but dramatically change which onion pages the upstream engines return. > > Or the reverse. > > So I would avoid combining them into a single “Nemotron lift” number. > > A minimal record for every refinement run could be: > > ```plaintext > topic_id > raw_query > refined_query > actual_backend > actual_model > fallback_used > frozen_corpus_id # for replay experiments > > ``` > > The `actual_backend` field is particularly useful when the architecture has automatic Nemotron → local Qwen fallback. Otherwise both outputs can accidentally enter the analysis as simply `refined=true`, even though they came from different models. > > For the live test, Tor/search availability introduces another variable, so paired runs are probably enough initially: > > ```plaintext > half: > raw -> refined > > half: > refined -> raw > > ``` > > or alternate the order over repeated trials. > > Keep `engineHealth` with each arm. Then a “refined query won” result is distinguishable from: > > > three extra engines happened to be reachable during the refined run. > > The 7B decision tree then becomes quite simple: > > ```plaintext > Nemotron has no reliable lift > -> do not spend time imitating it yet > > Lift only on frozen ranking > -> local model target = ranking-oriented query representation > > Lift mainly on live acquisition > -> local model target = engine-facing query rewriting / recall > > Lift in both > -> stronger case for a local 7B/8B replacement > > ``` > > This keeps the original goal — removing the remote-model dependency — but lets the benchmark tell you what behavior the local model actually needs to learn. > **MCP stdio: I would test semantic parity rather than code-path parity** > > For standalone MCP use, yes, I would update it. > > But I would define success as: > > ```plaintext > REST(query, fixture) > and > MCP-stdio(query, fixture) > > produce equivalent search semantics > > ``` > > rather than requiring the transport implementations themselves to look identical. > > A tiny deterministic fixture is probably enough. > > For example, mock three engines: > > ```plaintext > engine_a: > URL X, title/snippet A > URL Y > > engine_b: > URL X, different title/snippet > URL Z > > engine_c: > timeout > > ``` > > Then assert both interfaces agree on: > > ```plaintext > prepared query / URL behavior > engine status > raw candidate count > canonical candidate identity > duplicate provenance > selected representative title/snippet > ranking > Top-k > error / partial-success semantics > > ``` > > That test becomes much more useful than manually checking Claude Desktop after every ranking change. > > It also protects the architectural split you now have: > > ```plaintext > IntelShed > -> REST > > Claude Desktop / VS Code / MCP agents > -> stdio > > both > -> same Ghost search semantics > > ``` > > There is one transport-specific smoke test I would definitely add: **stdio purity**. > > The MCP transport specification explicitly reserves stdout for MCP JSON-RPC messages; server logging belongs on stderr. See the official [MCP stdio transport specification](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports). > > So a very cheap CI test is: > > ```plaintext > launch MCP server over stdio > perform initialize + ghost_search fixture call > > assert: > every stdout frame parses as MCP/JSON-RPC > > allow: > arbitrary diagnostic text on stderr > > ``` > > That catches an annoying class of integration failure that may never appear through the REST path. > > I would treat MCP protocol-version migration as a separate concern from search parity. If the SDK/protocol version changes later, the parity fixture should continue to pass regardless. > **Two small measurement boundaries I would keep for privacy/source claims** > > ### Tor stream isolation > > The current SOCKS credential approach has a real mechanism behind it. > > Tor’s current [SOCKS extensions specification](https://spec.torproject.org/socks-extensions.html) defines stream isolation based on SOCKS authentication values. Different isolation values can prevent streams from sharing circuits under the configured isolation rules. > > I would only change the wording slightly: > > ```plaintext > different per-search SOCKS isolation values > -> evidence that the intended isolation mechanism is being used > > observed StreamID -> CircuitID mapping > -> evidence that the actual searches were attached to different circuits > > ``` > > The second is stronger. > > If audit-grade evidence becomes useful, Tor’s [Control Protocol events](https://spec.torproject.org/control-spec/replies.html) expose circuit/stream information, including circuit IDs and SOCKS authentication metadata. A test-only controller can therefore observe two searches and record what circuits they actually used. > > That does not need to block the retrieval benchmark. > > It is just a clean way to keep: > > > architecture/configuration claim > > separate from: > > > runtime-observed privacy claim. > > One forward-looking detail: current Tor specifications call arbitrary SOCKS username/password isolation values **legacy isolation** and recommend the newer `` isolation format for new clients. I would not make migration a benchmark prerequisite, but it is worth putting on the privacy-audit list. > > ### Per-engine timeout values > > The per-engine timeout idea itself makes sense for heterogeneous onion services. > > I would describe the particular 20/30/40s values as **empirically tuned operational settings** unless there is a specific Tor Project source for the quoted generic “60–90 second” recommendation. > > Tor’s specifications document onion-service timeout/failure semantics, but I have not found a current generic client recommendation saying application connect timeouts should universally be 60–90 seconds. > > That is mostly a citation/wording boundary rather than an architectural issue. > **Frozen benchmark provenance: one cheap addition that will help collaborators later** > > Since the frozen corpus is becoming a real research artifact, I would put a tiny manifest beside it. > > For example: > > ```json > { > "corpus_version": "ghost-serp-v1", > "collected_at": "...", > "git_commit": "...", > "query_set_sha256": "...", > "engine_registry_sha256": "...", > "acquisition_config_sha256": "...", > "normalization_version": "...", > "tor_version": "...", > "result_count": 28280, > "snippet_coverage": 0.994 > } > > ``` > > The exact fields do not matter much. > > The important thing is that six months later somebody can tell whether: > > ```plaintext > same qrels + different ranking code > > ``` > > is really the experiment being run, rather than: > > ```plaintext > same filename + subtly different corpus semantics. > > ``` > > The qrels should then reference a stable `candidate_id` from that corpus rather than positional rank. > > That also makes the eventual dataset much easier to version if you later create: > > ```plaintext > v1 = SERP-surrogate relevance > v2 = larger query set > v3 = page-level judgments > > ``` > > without invalidating the earlier benchmark. So if I were choosing the shortest path from the current state, I would probably do: ```plaintext 1. Write query description/narrative for 4–6 topics. 2. Build a diverse Top-k pool. 3. Double-annotate that pilot. 4. Freeze rubric + candidate identity + duplicate handling. 5. Annotate the remaining topics. 6. Run atomic engine-count / BM25 / agreement / plain-RRF / trust comparisons. 7. Only then interpret the existing Jaccard differences as retrieval quality differences. 8. Split Nemotron into frozen-ranking and live-acquisition experiments. 9. Add the small REST/stdio parity fixture. ``` The Tor audit and heavier neural reranking can stay orthogonal to that path. That would preserve the direction you already have — Ghost as the acquisition/result-merging broker, IntelShed as the downstream intelligence layer — while making each subsequent number answer one fairly specific question. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 27, 2026, 5:44am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/5 "2026-08-27T05:44:35Z")
Thanks for the continued guidance. I’ve been working through your calibration recommendations and have concrete results to report, though I want to be clear upfront: most of this is still in progress, the qrels are LLM-generated and provisional, and I don’t yet know whether any of these numbers will hold up under scrutiny. This is a work-in-progress report, not a results claim. **On your calibration sequence (posts 2 and 4) — where things stand:** | Step | Your recommendation | Status | Result | | --- | --- | --- | --- | | 1 | Define information need per query | Done | 24 TREC-style narratives (title, description, narrative with grade examples) | | 2 | Diverse pool, not single ranker | Done | 4 arms: RRF, engine-count, BM25, unique-contribution. 1197 entries, avg 49.9/query | | 3 | Double-annotate 4-6 queries | Done | 6 queries, 289 pairs, 75.1% raw / 87.9% corrected agreement | | 4 | Freeze rubric after pilot | Done | q18 narrative refined, 0-vs-1 boundary clarified, commercial pages = grade 0 | | 5 | Annotate remaining topics | In progress | Running now, see below | | 6 | Atomic replay arms | Done | 8 strategies evaluated | | 7 | REST/stdio parity fixture | Done | 9 tests, 220/220 pass | | 8 | Provenance manifest | Done | SHA-256 hashes for corpus, queries, qrels, narratives, dedup report | I followed your ordering almost exactly. The one place I deviated was running the atomic ablations (step 6) before the full annotation (step 5) was complete — they don’t depend on qrels, so I used the existing heuristic qrels as a provisional baseline. The ablation numbers should be re-validated against the new qrels once they’re ready. **Calibration pilot — what we actually found:** 289 pairs across 6 queries (q03, q09, q14, q16, q18, q22), judged by Nemotron-3-Super-120B and DeepSeek V4 Flash with the formalized narratives: - Raw agreement: 75.1% (217/289) - Corrected agreement (excluding 17 Nemotron infrastructure failures): 87.9% (239/272) - Cohen’s kappa: 0.5808 unweighted, 0.7502 weighted - 0-vs-2 boundary is stable: only 2 extreme disagreements in 289 pairs - Disagreements cluster at the 0-vs-1 and 1-vs-2 boundaries, which is expected for graded relevance The 17 excluded pairs were Nemotron 503/timeout errors that defaulted to a parse-fallback grade. This was a bug — I’ll come back to it below. q18 (“investigative journalism”) was the worst at 58% agreement. The narrative was too ambiguous about forum/directory results. I refined it: forums and directories listing journalism resources are grade 1 (peripheral) unless they provide direct operational access (SecureDrop instances, Signal contacts, encrypted email). I also clarified that commercial “hire a hacker” pages and exploit listings are grade 0 — the research intent is analysis, not purchasing. I want to be honest that 87.9% corrected agreement is not the same as 87.9% accuracy. Both judges could be wrong in the same way. The agreement tells us the rubric is interpretable; it does not tell us the judgments are correct. **Judgment pool — diversity check:** The 4 arms produced 1197 entries after dedup. The interesting number is Arm D (unique-contribution): it surfaces 430 documents that no other arm finds. These are long-tail URLs that agreement-based rankers (RRF, engine-count) structurally cannot surface. Whether any of them are actually relevant is an open question — but at least they’re in the pool now, which was your main point about pooling bias. I have not yet computed the pairwise engine overlap matrix you suggested in Phase 1. The frozen corpus has the raw per-engine data, but I prioritized the qrels pipeline. It’s on the list. **Atomic ablations — provisional, using heuristic qrels:** | Strategy | nDCG@10 | Delta vs baseline | | --- | --- | --- | | rrf-plain (baseline) | 0.4697 | — | | rrf-trust | 0.5899 | +0.1203 | | rrf-quality | 0.5266 | +0.0569 | | rrf-trust-quality (production) | 0.5669 | +0.0973 | | engine-count | 0.3706 | -0.0990 | | agreement-only | 0.2787 | -0.1910 | | bm25 | 0.4203 | -0.0494 | Trust weighting is the largest single contributor. Pure engine agreement is the worst strategy. BM25 underperforms RRF on this corpus. These numbers are provisional — they use heuristic qrels, not the calibrated ones. I’ll re-run them once the full annotation is complete. **Full re-annotation — in progress, with a complication I want to be transparent about:** I’m running two parallel annotation passes: - **3-judge ensemble** (DeepSeek V4 Flash, gpt-oss-120b, Nemotron-3-Super-120B): majority vote \>=2 of 3, DeepSeek as tiebreaker. This is about 50% complete (q01-q12 done, q13-q24 running). - **5-judge ensemble** (the above three + GLM-5.2 and Kimi-K3): majority vote \>=3 of 5. This just started. The reason for the second run is something I think is worth raising, because it may be relevant to other people building LLM-judged benchmarks in security-adjacent domains. During the 3-judge run, I discovered that a significant fraction of model responses were empty or unparseable, and the parser was silently converting these to grade 1. DeepSeek had ~18% parse-fallbacks, gpt-oss ~22%, Nemotron ~7%. This is a systematic bias toward grade 1 that has nothing to do with relevance — it’s a parser bug. I fixed it (empty/unparseable responses now trigger retries, and after retries are exhausted they’re recorded as errors rather than assigned a grade), but the 3-judge qrels produced before the fix contain this contamination. The 5-judge run uses the fixed parser. I’ll compare the two once both are complete, and the difference should quantify how much the parser bug affected the results. **On the ensemble composition — a methodological question I’m still thinking through:** The 5-judge ensemble has 3 Chinese-origin models (DeepSeek, GLM-5.2, Kimi-K3) and 2 Western-origin models (gpt-oss, Nemotron). This wasn’t accidental. There’s published evidence that Western models exhibit what’s been called “Defensive Refusal Bias” — systematic downgrading of security research content due to keyword-based safety filters. A peer-reviewed study of 2,390 prompts from the National Collegiate Cyber Defense Competition found LLMs refuse defensive security requests at 2.72x the rate of semantically equivalent neutral requests (arXiv 2603.01246). The HuggingFace incident in July 2026 — where HuggingFace’s incident responders had to switch to GLM-5.2 because their primary Western LLM refused to analyze the breach — is a concrete example. For a benchmark where all 24 queries are security-adjacent, this matters. If gpt-oss and Nemotron systematically downgrade SecureDrop pages or exploit writeups, a 2-Western-1-Chinese ensemble would produce systematically lower grades for exactly the content the benchmark is meant to serve. But I want to be careful here. The evidence supports the mechanism (Western safety filters downgrading security content). It does not directly prove that my specific 3-Chinese-2-Western composition produces better qrels. That’s an empirical question I can only answer by comparing the 3-judge and 5-judge results. I’ll report whatever the comparison shows, including the possibility that the composition doesn’t matter for this particular benchmark. I’d welcome your thoughts on whether this reasoning is sound or whether I’m overcorrecting. **What I have not done yet (being honest about the gaps):** - Nemotron query refinement lift: not measured yet. Your split into frozen-ranking test vs live-acquisition test is noted and I think it’s the right design. It’s next after the qrels stabilize. - stdio purity test (stdout = MCP JSON-RPC only): not implemented. You’re right that this is a separate class of failure that the REST path won’t catch. - Tor stream isolation: I have SafeSocks 1 and per-search SOCKS credential isolation, but I have not done the ControlPort observation you described. The distinction between “configured” and “observed” is fair. - Per-engine timeout citation: I described 60-90s as a “Tor Project recommendation” but I don’t have a specific source for that. I’ll either find the citation or reword it as empirically tuned. - judged@10 / judged@20 tracking: not yet implemented. I’ll add this before evaluating any new retrieval system against the qrels. - Pairwise engine overlap matrix: data is available, computation is not done yet. - bpref / incomplete-judgment evaluation: not needed yet, but I’ll keep it in mind if new systems retrieve mostly unjudged documents. **On the “first” claim:** You mentioned Morato et al. already evaluated retrieval/ranking across Tor search engines. I’ve removed “first” from my descriptions. I haven’t found a reusable public onion query-qrels collection equivalent to what I’m building, but I also haven’t done an exhaustive literature search. I’d rather under-claim and be corrected than over-claim and be wrong. **What I expect to report next:** Once both annotation runs complete, I’ll have: - 3-judge qrels (with parser contamination, documented) - 5-judge qrels (with fixed parser) - Per-doc comparison between the two - Grade distribution shift - IR metric impact (nDCG@10, MAP, Recall with both qrels versions) - System ranking stability (do the relative rankings of RRF/BM25/engine-count change?) If the two qrels sets produce the same system rankings, that’s the strongest evidence the benchmark is robust. If they don’t, I need to understand why before claiming anything. All qrels will be explicitly labeled as LLM-generated and provisional, not human-validated ground truth. I’m one researcher with LLM judges — that’s a methodological limitation, not a feature. Thanks again -John6666- for the structured feedback. The calibration sequence you proposed in post 4 shaped the entire pipeline. I’m happy to share any of the raw data (narratives, pool, per-judge reasoning logs) if it’s useful for your own work. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 27, 2026, 11:34pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/6 "2026-08-27T23:34:37Z")
Quick update — both annotation runs are complete and most of the gaps I listed in my previous post are now closed. This is a final results report, not another work-in-progress. A few things changed since my last post that I want to be explicit about, because they affect interpretation of the numbers below: 1. **\*\*Production ranking changed from engine-count to RRF + quality filter.\*\*** In my post 3 I described engine-count as the production default. During the benchmark cycle, the ablations showed RRF with trust weighting and quality filtering consistently outperformed both engine-count and BM25. I switched the production path to RRF + quality filter (RRF k=10, trust-weighted per engine, then quality filter applied). The benchmark numbers below reflect this — “rrf-quality” is the current production configuration, not the one I described in post 3. 2. **\*\*RRF k changed from 60 to 10.\*\*** My post 3 mentioned k=60 following Cormack et al. SIGIR 2009. After empirical tuning on the frozen corpus, k=10 produced better top-10 rankings for this domain. The Cormack paper used k=60 on TREC newswire; onion SERPs have shorter result lists and different rank-depth distributions, which favors a smaller k. The benchmark numbers below all use k=10. 3. **\*\*Engine count: 13, not 14.\*\*** My post 1 said 14 engines. Torgol was removed during the benchmark cycle (transient Tor circuit issues, not a parser bug — it returned results intermittently). The frozen corpus and all benchmark numbers below use 13 engines. The engine overlap matrix is 13×13. 4. **\*\*Quality filter explanation.\*\*** Several posts refer to “rrf-quality” without explaining what the quality filter does. It’s a 6-strategy demotion pipeline that never removes results — it only reorders: (1) domain collapsing (max N per domain), (2) SimHash near-duplicate detection (Hamming distance ≤3), (2b) semantic near-duplicate via embeddings (cosine \>0.85), (3) information-quality scoring (snippet/entity/title signals), (4) spam/mirror detection (same title, different domain), (5) commercial-intent penalty (buy/shop/price keywords demoted for research queries), (6) URL-path mirror network detection. The filter is OFF by default and must be explicitly enabled. “rrf-quality” in the benchmark = RRF ranking followed by quality filter reordering. 5. **\*\*Test count: 223, not 180.\*\*** My post 1 and 3 said 180 tests. The current suite has 223 tests (220 pass, 3 stdio purity tests are environment-dependent and time out without a running MCP server process). **## Calibration sequence — final status** All 8 steps are now complete. The two that were in progress last time: | Step | Previous status | Final status | |------|----------------|--------------| | 5 — Full re-annotation | “~50% complete” | **\*\*Done\*\*** — 24/24 queries, 1197 pairs, both 3-judge and 5-judge | | 6 — Atomic ablations | “provisional, heuristic qrels” | **\*\*Re-validated\*\*** against both qrels\_v2 and qrels\_v3 | **## Two complete qrels sets** I now have two independent qrels for the same 1197-pair judgment pool: **\*\*3-judge (qrels\_v2):\*\*** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B. Majority \>=2 of 3, DeepSeek tiebreaker. This is the run that had the parser-fallback contamination in q01-q07. I kept it as-is rather than re-running, because the comparison itself is informative. **\*\*5-judge (qrels\_v3):\*\*** DeepSeek V4 Flash + gpt-oss-120b + Nemotron-3-Super-120B + Llama-3.3-70B + Kimi K2.6. Majority \>=3 of 5, DeepSeek tiebreaker. Fixed parser. This is the canonical qrels going forward. One composition change since last post: I replaced GLM-5.2 with Llama-3.3-70B as the 4th judge. I ran a 25-document head-to-head test (5 queries, same prompts, alternating): | Model | Exact match | Parse failures | Avg latency | |-------|------------|----------------|-------------| | Llama-3.3-70B | 24/25 (96%) | 0 | 1138ms | | GPT-4.1-Nano | 18/25 (72%) | 0 | 796ms | | GLM-5.2 | 15/25 (60%) | 8 | 1207ms | | Qwen3.8-Flash | 1/10 (10%) | 5 | 5453ms | | Mistral-Large-3 | 0/25 | 25 | unavailable | GLM-5.2 had 8 parse failures in 25 calls and only 60% exact match. Llama-3.3-70B had 0 parse failures and 96% match. The full 5-judge run confirmed this: Llama achieved 100% coverage over 872 documents with QWK 0.79 against DeepSeek. The replacement was the right call. I also switched Kimi-K3 to Kimi K2.6 mid-run (73% cost reduction, parser already handles code-fences). This is a model-version change, not a model-family change. **## 3-judge vs. 5-judge comparison** This was the empirical question I raised last time — whether the ensemble composition actually matters for this benchmark. | Metric | Value | |--------|-------| | Per-doc agreement | 89.9% (1076/1197) | | Disagreement | 10.1% (121) | Grade shift matrix (3j rows × 5j cols): ``` ``` | 5j:0 5j:1 5j:2 ``` -------|----------------- 3j:0 | 709 40 8 3j:1 | 18 249 10 3j:2 | 7 38 118 ``` The 5-judge ensemble is more conservative at Grade 2 (-27 docs) and more generous at Grade 1 (+50 docs). The biggest shift is 3j=0 → 5j=1 (40 docs) — these are documents the 3-judge called irrelevant that the 5-judge calls peripheral. The reverse shift (3j=1 → 5j=0, 18 docs) is smaller. I attribute the Grade-2 conservatism partly to Kimi, who votes Grade 1 for 45.6% of all documents and only assigns Grade 2 at 1.8%. The other four judges assign Grade 2 at 8-16%. Kimi is the weakest ensemble member by pairwise agreement: | Pair | QWK | Landis-Koch | |------|-----|-------------| | deepseek ↔ llama | 0.7901 | substantial | | deepseek ↔ gpt-oss | 0.7753 | substantial | | gpt-oss ↔ nemotron | 0.7620 | substantial | | gpt-oss ↔ llama | 0.7425 | substantial | | deepseek ↔ nemotron | 0.7361 | substantial | | nemotron ↔ llama | 0.7322 | substantial | | llama ↔ kimi | 0.3383 | fair | | nemotron ↔ kimi | 0.2919 | fair | | gpt-oss ↔ kimi | 0.2810 | fair | | deepseek ↔ kimi | 0.2574 | fair | All non-Kimi pairs are “substantial” (0.61-0.79). All Kimi pairs are “fair” (0.23-0.34). Kimi is a candidate for replacement in the next iteration. I have not replaced it yet because the ensemble decisions are robust to a single weak judge — in all 1197 pairs, the other 4 judges always had \>=3 agreement, so Kimi’s vote never changed a majority outcome. **## Ranking stability — the key question** The most important result: **\*\*the system ranking is identical on both qrels versions.\*\*** | Strategy | v2 nDCG@10 | v3 nDCG@10 | Delta | |----------|-----------|-----------|-------| | engine-count | 0.2607 | 0.2762 | +0.016 | | bm25 | 0.2864 | 0.2915 | +0.005 | | rrf | 0.3219 | 0.3592 | +0.037 | | **\*\*rrf-quality\*\*** | **\*\*0.3930\*\*** | **\*\*0.4566\*\*** | **\*\*+0.064\*\*** | Order on both: rrf-quality \> rrf \> bm25 \> engine-count. The 5-judge qrels lift rrf-quality by +16.3% (0.3930 → 0.4566) because Llama and DeepSeek identify more Grade-2 documents than GLM did (16.1% vs 3.1% Grade-2 rate). But the relative order of strategies doesn’t change. This is the strongest evidence I have that the benchmark is robust — two independent judge ensembles with different compositions and a parser-fix in between produce the same ranking conclusion. **## Atomic ablations — re-validated against both qrels** The ablation numbers from last post used heuristic qrels. I’ve now re-run them against both LLM-judge qrels. The relative ordering of components is consistent: - Trust weighting remains the largest single contributor (+0.12 nDCG over plain RRF) - Quality filter adds +0.057 nDCG independently - Pure engine agreement remains the worst strategy - BM25 underperforms RRF on this corpus The absolute numbers shifted because the qrels changed, but the component ranking didn’t. Trust + Quality (the production configuration) wins on both qrels versions. **## External validation against NIST human qrels** This is new work I didn’t mention last time. I found a public dataset that allowed me to validate my judges against human ground truth: asukul’s RAG-Eval-LLM-Judge repository (Adisak Sukul, Iowa State University), which releases 537 stratified-balanced TREC RAG 2024 pairs with NIST human relevance labels. I ran a 15% stratified sample (80 pairs) through my 5-judge ensemble: | Judge | kappa vs NIST | QWK vs NIST | Coverage | Errors | |-------|--------------|-------------|----------|--------| | DeepSeek V4 Flash | 0.4095 | 0.5833 | 100% | 0 | | GPT-OSS-120B | 0.3302 | 0.4419 | 100% | 0 | | Llama-3.3-70B | 0.3239 | 0.5484 | 100% | 0 | | Kimi K2.6 | 0.3091 | 0.4710 | 100% | 0 | | Nemotron-3-Super | 0.2760 | 0.4250 | 100% | 11 | | 5-judge ensemble | 0.3458 | 0.5090 | 100% | 11 | For context, asukul reports a 9-judge ensemble kappa of 0.4941 against the same NIST qrels, and their DeepSeek V4 Pro achieved kappa=0.4705 but with only 39% coverage (212/537 valid responses). My DeepSeek V4 Flash (kappa=0.4095, 100% coverage) has overlapping confidence intervals with their DeepSeek V4 Pro (kappa=0.4705, 39% coverage). The coverage difference is the more practically significant finding — V4 Flash returns valid responses for every pair, V4 Pro fails on 61%. My ensemble kappa (0.3458) is below asukul’s (0.4941). I attribute this to three factors, in approximate order of magnitude: 1. **\*\*Domain mismatch.\*\*** My judge prompt is calibrated for onion-network security content (journalists, investigators, security researchers). asukul’s prompt is 4 lines of generic RAG instructions. On academic TREC queries, my domain-specific calibration hurts — the “commercial pages = grade 0” rule and “intent fit = security research” criterion shift thresholds in ways that don’t match NIST assessor behavior on academic topics. 2. **\*\*Scale collapse.\*\*** asukul uses a 4-grade scale (0,1,2,3). I use a 3-grade scale (0,1,2) and collapse 2+3 → 2. This loses information exactly where my judges are weakest — they correctly identify Grade 3 documents 85% of the time but Grade 2 documents only 45% of the time. 3. **\*\*Ensemble size.\*\*** 5 judges vs. 9 judges. More judges reduce variance, though the effect is smaller than domain and scale. I did not run the full 537 pairs because the domain mismatch is structural — more data would confirm the gap, not close it. The validation was worth doing to confirm that my judges produce reasonable kappa against human ground truth (0.41 for DeepSeek is “moderate” agreement, not random), but I would not use TREC RAG as a primary validation target for an onion-search benchmark. **## Nemotron error analysis** I mentioned last time that Nemotron had infrastructure failures. In the full 5-judge run: 85/1197 errors (7.1%), all HTTP 429 (NVIDIA NIM rate limiting). 10 of 11 errors in the TREC validation run were on high-relevance pairs (human grade 2 or 3). The error latency pattern is consistent with rate limiting: Nemotron’s OK responses average 981ms, error responses average 1125ms — the request goes out, waits, then gets rejected. On my onion corpus (longer prompts, slower throughput), Nemotron had 0 errors in 509 documents during the initial run. The TREC corpus has shorter passages and faster request rates, which triggers the rate limit. Ensemble impact: zero. In all 85 error cases, the other 4 judges had \>=3 agreement, so Nemotron’s fallback vote (relevance=1) never changed a majority outcome. The 5-judge majority design is robust to single-judge failure, but this is luck as much as design — if two judges failed simultaneously on the same document, the remaining 3 would decide, and the fallback votes could matter. **## Items from the “not done yet” list** All closed except the Nemotron lift experiment: | Item | Previous | Now | |------|----------|-----| | Nemotron query refinement lift | “not measured” | **\*\*Partially measured, not with your design.\*\*** I ran a frozen-ranking test on 2026-08-25 using heuristic qrels (not the calibrated LLM-judge qrels): nDCG -0.0015, MAP -0.0008, MRR -0.0139 — no lift, 23/24 queries unchanged. But this was only the frozen-ranking half, with provisional qrels. Your full design — frozen-ranking test + live-acquisition test, with calibrated qrels — has not been run. It’s the first item in the post-benchmark phase. | | stdio purity test | “not implemented” | Done — 3 tests, stdout = JSON-RPC only | | Tor stream isolation | “configured, not observed” | Still configured-only; ControlPort observation deferred | | Per-engine timeout citation | “don’t have source” | Done — Tor spec `cbtinitialtimeout=60s` + Tor Stack Exchange q/21966 | | judged@10 / judged@20 | “not yet implemented” | Done — rrf=1.0, rrf-quality=0.892 (quality filter pushes some unjudged docs into top-10) | | Pairwise engine overlap matrix | “not done” | Done — 13×13 Jaccard, only excavator↔oss=0.55 correlated, ahmia 97.9% unique contribution | | bpref / incomplete-judgment | “not needed yet” | Still not needed — judged@10 \>=0.89 for all strategies | **## What I’m taking from this** The benchmark answered the question I needed it to answer: **\*\*which ranking strategy should production use, and is that answer robust to judge-ensemble composition?\*\*** The answer is rrf-quality, and yes, it’s robust — the same strategy wins on both a 3-judge qrels with a known parser bug and a 5-judge qrels with the fix and a different ensemble composition. The external validation against NIST human qrels confirmed that my judges produce moderate agreement with human assessors (DeepSeek kappa=0.41), which is not validation of correctness but is validation of non-randomness. The gap to asukul’s 9-judge ensemble (0.49) is attributable to domain and scale differences, not judge quality. The qrels remain LLM-generated and provisional. I have not done human review of any of the 1197 pairs, and I won’t claim correctness I haven’t measured. What I can claim is that the ranking conclusion is stable across two independent judge ensembles, and that the judges produce moderate agreement with external human ground truth on an out-of-domain benchmark. The next phase is production features — applying the benchmark findings to the actual search system. The benchmarking cycle is complete, with one exception: the Nemotron query refinement lift experiment you requested in post 4 (frozen-ranking test + live-acquisition test with calibrated qrels) has only been partially run — a frozen-ranking test with heuristic qrels showed no lift, but the full design with LLM-judge qrels and the live-acquisition arm is still open. It’s the first item on the post-benchmark roadmap. Thanks again for the calibration sequence. It shaped the pipeline from start to finish, and the ranking-stability result is directly attributable to your emphasis on diverse pooling and double-annotation before full re-annotation. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 28, 2026, 12:53am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/7 "2026-08-28T00:53:16Z")
I think the concern is warranted: * * * My short answer would be: **the motivation for broadening the judge panel is sound, but I would treat “3 Chinese-origin + 2 Western-origin” as an initial diversity heuristic, not as the validity argument itself.** The stronger argument is whether those five judges, on _this benchmark_, actually have complementary failure modes and produce more stable retrieval conclusions. There is good evidence for the mechanism you are worried about. Campbell et al.'s [Defensive Refusal Bias](https://arxiv.org/abs/2603.01246) study used 2,390 real NCCDC examples and found that defensive requests containing security-sensitive language were refused at **2.72×** the rate of semantically equivalent neutral requests. The July 2026 [Hugging Face incident disclosure](https://huggingface.co/blog/security-incident-july-2026) is also a useful real-world example: HF says the commercial API models they initially tried blocked substantial forensic analysis of real attack commands/exploit/C2 material, so they moved the analysis to self-hosted GLM-5.2. Their [technical timeline](https://huggingface.co/blog/agent-intrusion-technical-timeline) gives more detail. But neither result establishes: ```plaintext Chinese-origin judge -> more accurate relevance labels Western-origin judge -> systematically worse relevance labels 3 Chinese + 2 Western -> optimal panel ``` They establish a **failure mechanism worth controlling for**. I would therefore keep the heterogeneous five-judge panel, but measure the judges as instruments rather than using origin as the proxy for validity. The highest-information path from where you are now seems to me to be: ```plaintext old original-3 + contaminated parser ↓ same original-3 + fixed parser ↓ all-5 + fixed parser ↓ all 3-of-5 subsets + all 4-of-5 LOJO panels ↓ small stratified human anchor ↓ winner / pairwise system-order stability ↓ stabilized qrels → rerun ranking ablations ``` The first distinction is especially important: ```plaintext old-3 vs fixed-3 = parser/error-semantics effect fixed-3 vs fixed-5 = panel expansion/composition effect ``` Comparing the contaminated 3-judge qrels directly with the fixed-parser 5-judge qrels is still useful as an end-to-end before/after result, but it cannot by itself quantify how much came from the parser fix versus adding GLM/Kimi. If you retain the five **raw per-judge labels/statuses** , almost everything after the fixed five-judge run is cheap offline analysis rather than more inference. > **Judge composition: I would measure the judges, not the passport category** > > ### 1. First deconfound parser repair from panel expansion > > I would explicitly preserve three qrel versions: > > ```plaintext > A = original 3 judges + original parser > B = original 3 judges + fixed parser > C = all 5 judges + fixed parser > > ``` > > Then: > > ```plaintext > A -> B > measures: > parser fallback / retry / infrastructure-error contamination > > B -> C > measures: > additional judges + changed aggregation behavior > > ``` > > That distinction matters here because the old parser was not injecting random noise. Mapping empty/unparseable responses to **grade 1** creates a directional bias toward the middle relevance grade. > > So I would treat these separately in the artifact: > > ```plaintext > valid grade > refusal > empty response > API/provider error > timeout > parse failure > retry exhausted > > ``` > > and never convert the last five into relevance grades. > > A missing judge output is an **availability/policy/infrastructure observation** , not evidence that the document is marginally relevant. > > * * * > > ### 2. Once the five labels exist, enumerate the panel space offline > > With five fixed-parser raw judge outputs, you can generate: > > ```plaintext > all-five panel > > C(5,3) = 10 different 3-of-5 panels > > 5 different 4-of-5 leave-one-judge-out panels > > ``` > > without another model call. > > For each panel I would compare at least: > > ```plaintext > qrel flip rate > 0↔1 changes > 1↔2 changes > extreme 0↔2 changes > unresolved / insufficient-quorum count > grade distribution > nDCG@10 / MAP / Recall > winner > top-2 ordering > every pairwise retrieval-system order > > ``` > > The last few are probably more useful than raw label agreement alone. > > For example, suppose two panels disagree on 15% of individual document grades, but both produce: > > ```plaintext > RRF-trust > > > RRF-plain > > > BM25 > > > engine-count > > ``` > > for essentially every system comparison. That is a substantially more reassuring result for a **retrieval benchmark** than 95% document-level agreement accompanied by a winner flip. > > Conversely, if removing one judge changes the winner or several pairwise system comparisons, that judge deserves direct inspection even if the overall five-judge agreement coefficient looks good. > > This is also where leave-one-judge-out becomes very interpretable: > > ```plaintext > remove judge J > ↓ > how many qrels change? > how many retrieval-system pairwise orders change? > does the winner change? > > ``` > > If one judge has disproportionate influence, audit that model. > > If all ten 3-of-5 subsets and most 4-of-5 panels produce essentially the same retrieval conclusions, I would probably **stop optimizing the geographic ratio**. At that point the empirical result is stronger than the taxonomy. > > * * * > > ### 3. “Diversity” should mean diversity of error, not diversity of labels > > There is independent work supporting multi-judge relevance assessment in general. For example, [JudgeBlender](https://arxiv.org/abs/2412.13268) explicitly ensembles multiple LLMs or multiple prompts to reduce single-judge bias and obtains competitive automatic relevance assessments. > > That supports the _ensemble instinct_. > > But the useful diversity variable is something like: > > ```plaintext > pairwise judge disagreement > shared human-label error > refusal/error correlation > grade-distribution correlation > terminology sensitivity > LOJO influence > > ``` > > rather than: > > ```plaintext > country-of-origin count > > ``` > > because two models from different organizations/countries can still share the same failure, while two models from the same broad region may behave quite differently. > > A table like this would answer your methodological question much more directly: > > | Judge | Parse/error % | Refusal % | Mean grade | Human agreement | Security-wording delta | LOJO system-order effect | > | --- | --- | --- | --- | --- | --- | --- | > | DeepSeek | | | | | | | > | gpt-oss | | | | | | | > | Nemotron | | | | | | | > | GLM | | | | | | | > | Kimi | | | | | | | > > Then “why these five?” has an empirical answer. > > * * * > > ### 4. I would add one tiny paired wording control > > Because the hypothesis is specifically **defensive refusal / safety-language sensitivity** , you can test the mechanism much more directly than by comparing national origins. > > For a small subset of hand-checked cases, prepare semantically equivalent formulations: > > ```plaintext > security-sensitive wording > vs > neutral wording expressing the same defensive information need > > ``` > > and send both to every judge with the same result surrogate. > > Measure separately: > > ```plaintext > refusal / empty / parse-error shift > grade shift > 0↔1 / 1↔2 boundary shift > reasoning change > > ``` > > You do not need to neutralize all 24 topics. A carefully checked sample of the categories you expect to stress safety behavior is enough to tell whether the mechanism is present in _your_ judge panel. > > That is close to the experimental logic of [Defensive Refusal Bias](https://arxiv.org/abs/2603.01246): hold intent approximately constant, alter the security-sensitive framing, and observe the change. > > Possible outcomes: > > ```plaintext > no meaningful judge-specific wording effect > -> geography probably is not doing useful work here > > one or two judges shift strongly > -> audit/weight/replace those judges > > all judges shift similarly > -> adding geographically diverse judges did not remove the shared failure > > different judges fail on different examples > -> heterogeneous ensemble may genuinely be buying robustness > > ``` > > This is, to me, a cleaner answer to the concern than trying to infer safety behavior from model provenance. > > * * * > > ### 5. Predeclare quorum/tie semantics too > > With three judges, `2/3` majority is easy until all three choose different ordinal grades. > > With five judges, distributions like: > > ```plaintext > 2 votes grade 0 > 2 votes grade 1 > 1 vote grade 2 > > ``` > > have no strict `3/5` majority. > > And once errors/abstentions are allowed, the denominator itself can change. > > I would therefore freeze: > > ```plaintext > minimum number of valid judges > missing/error semantics > tie semantics > insufficient-quorum semantics > > ``` > > before comparing panel variants. > > For graded `0/1/2` relevance, one useful sensitivity baseline is simply the **ordinal median** of valid judges, because it avoids giving a named tiebreaker extra influence. I would not necessarily replace your majority rule with it, but comparing: > > ```plaintext > majority + unresolved > vs > ordinal median > > ``` > > is almost free. > > If an item lacks sufficient valid judges, I would make it: > > ```plaintext > unjudged / audit queue > > ``` > > rather than manufacture a relevance label. > **Small human anchor + system-level validity** > > I think your sentence > > > agreement is not accuracy > > is exactly the right boundary. > > I would now resist the temptation to solve that by adding still more automatic judges. The next highest-information addition is a **small human anchor** , not full human annotation. > > ### 1. Human-label the cases where human information has the most value > > Something on the order of tens of examples can already be informative; if 50–100 is feasible, even better. > > I would stratify rather than sample only uniformly: > > ```plaintext > judge disagreements > 0-vs-1 boundaries > 1-vs-2 boundaries > previous refusal/error/parser cases > security-sensitive wording cases > q18 / SecureDrop / journalism edge cases > unique-contribution pool documents > cases where one judge is the LOJO pivot > random unanimous/easy cases as controls > > ``` > > If possible, have humans blind to: > > ```plaintext > which ranker produced the document > which judge gave which grade > which pool arm introduced it > > ``` > > so the anchor does not simply reproduce the automatic pipeline’s assumptions. > > You do not need humans to replace the LLM panel. > > [LARA: LLM-Assisted Relevance Assessments](https://arxiv.org/abs/2411.06877) is a useful conceptual reference here: use limited human judgments where they provide the most information, then use them to calibrate/debias cheaper automatic assessment rather than pretending the automatic labels are perfect ground truth. > > * * * > > ### 2. Judge the benchmark at two levels > > This distinction seems particularly important after [TREC 2025’s RAG track](https://trec.nist.gov/pubs/trec34/papers/Overview_rag.pdf). > > For its automatic Relevance Judgment task, even the best automatic submissions only achieved roughly **0.30–0.34 direct agreement fraction** with human relevance judgments. > > But elsewhere in the same evaluation, automatic assessments reproduced **run-level ranking behavior** much more strongly: aggregated retrieval-system scores had high Kendall rank agreement with manual evaluation even while individual/narrative-level judgments were noisier. > > That suggests two separate validity questions: > > ```plaintext > 1. label validity > How often do these qrels agree with a human anchor? > > 2. evaluation validity > Do these qrels lead us to the same conclusions about retrieval systems? > > ``` > > Ghost primarily needs the second property if the benchmark’s purpose is: > > > compare BM25 / RRF / trust / agreement / rerankers reproducibly. > > So for every qrel variant I would report both. > > ### Label-level > > ```plaintext > human agreement > weighted confusion matrix > per-grade precision/recall if sample size allows > judge-specific false-low / false-high behavior > > ``` > > ### System-level > > ```plaintext > Kendall / Spearman across all systems > winner preserved? > top two preserved? > all pairwise system-order flips > per-query source of each flip > > ``` > > With only a handful of ranking arms, I especially like the pairwise matrix because it is very hard to hide a practical instability inside a single correlation coefficient. > > For example: > > ```plaintext > qrels A qrels B human-anchor-corrected > RRF > BM25 yes yes yes > RRF > engine yes yes yes > BM25 > engine yes no yes > ... > > ``` > > That immediately tells the reader where the benchmark conclusion is fragile. > > * * * > > ### 3. A useful stop rule > > I would stop spending effort on judge-composition optimization when something like this is true: > > ```plaintext > fixed-3 and fixed-5 produce same winner > most/all 3-of-5 subsets preserve pairwise system order > LOJO does not expose one dominant judge > small human anchor finds no systematic panel failure > security-sensitive wording control does not expose a hidden directional bias > > ``` > > At that point “is 3 Chinese / 2 Western exactly right?” becomes much less important. > > You have directly shown that the result is not sensitive to reasonable judge-panel perturbations. > > If the system ordering _isn’t_ stable, then the raw per-judge matrix tells you where to look before investing in more retrieval machinery. > **Qrels and pooling: I would stabilize this before interpreting the ablation numbers strongly** > > The four-arm pool is a meaningful improvement, especially the **unique-contribution arm**. > > The fact that it contributes 430 documents not surfaced by the agreement/fusion arms is already useful because it shows why: > > ```plaintext > pool generator > > ``` > > and: > > ```plaintext > production ranker > > ``` > > do not have to be the same thing. > > The next question is simply: > > ```plaintext > how many of those 430 are relevant? > > ``` > > If an arm is mediocre as a production ranking strategy but contributes many **uniquely relevant** documents to the judgment pool, it is doing valuable benchmark work. > > I would eventually report: > > ```plaintext > unique candidates by arm > judged unique candidates > unique relevant candidates > unique relevant@equal-depth > > ``` > > rather than only raw unique count. > > * * * > > ### 1. Be careful about provisional qrels that favor their own pool contributor > > The classic problem is well documented in the IR literature. > > NIST’s [Bias and the Limits of Pooling for Large Collections](https://www.nist.gov/publications/bias-and-limits-pooling-large-collections) discusses how incomplete/shallow pools can create biased relevance sets, and [Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements](https://www.nist.gov/publications/reliable-information-retrieval-evaluation-incomplete-and-biased-judgements) explicitly notes that pooling is inherently capable of disadvantaging systems that did not contribute judged documents. > > The nuance is important: **pooling is not inherently unusable**. Well-designed, sufficiently deep and diverse TREC pools have often proved surprisingly reusable. The risk grows when: > > ```plaintext > the pool is shallow > the contributing systems are homogeneous > a new system retrieves many unseen documents > unjudged = nonrelevant > > ``` > > So I would not throw away the current results. > > I would simply make the fully judged four-arm pool the boundary between: > > ```plaintext > exploratory ablation > > ``` > > and: > > ```plaintext > evidence about component effectiveness > > ``` > > If the heuristic/provisional qrels used for the current table were heavily influenced by trust-weighted RRF candidates, then numbers such as: > > ```plaintext > rrf-trust +0.1203 > BM25 -0.0494 > > ``` > > are excellent hypotheses for what to re-test, but I would avoid interpreting the exact delta causally until the diverse pool is judged. > > Your planned: > > ```plaintext > judged@10 > judged@20 > > ``` > > is exactly useful here. > > For every evaluated system, I would report: > > ```plaintext > judged@10 > judged@20 > > ``` > > alongside nDCG/MAP whenever the system was not strongly represented in pool construction. > > If a new ranker looks worse while half its top 20 is unjudged, that is a different diagnosis from: > > > its judged top 20 is genuinely less relevant. > > A cheap additional robustness check is **leave-one-pool-arm-out** : see how much the resulting system comparisons depend on the unique judgments introduced by each pool mechanism. > > * * * > > ### 2. Freeze the information need, not just the query > > The 24 TREC-style narratives are a good move. > > There is now direct experimental support for this in [Formalized Information Needs Improve Large-Language-Model Relevance Judgments](https://arxiv.org/abs/2604.04140): LLM assessors given richer retrieval topics/descriptions/narratives show better agreement and more reliable relevance assessment than query-only assessors. > > This seems particularly important for short Ghost queries such as: > > ```plaintext > zero-day > secure drop > bitcoin > > ``` > > where two judges can apply the _same rubric_ while silently assuming different information needs. > > I would freeze and version: > > ```plaintext > query > description > narrative > grade examples > judge prompt > rubric > > ``` > > before the final annotation pass. > > And if narratives were initially drafted using examples from the current retrieval pool, I would simply record that provenance and make the reviewed/frozen narrative version the benchmark definition. The important thing is that the topic definition does not keep drifting after you start interpreting final ranker scores. > > * * * > > ### 3. Keep topical relevance separate from quality-policy attributes > > One design separation I would keep is: > > ```plaintext > relevance_grade > > ``` > > versus things such as: > > ```plaintext > commercial_intent > generic_directory > duplicate/mirror > quality_flag > unjudgeable > > ``` > > The reason is not that commercial pages or directories should rank highly. > > It is that their relevance can depend on the topic. > > For example, if the information need is investigating the existence or characteristics of an underground market, a marketplace listing can itself be relevant evidence even if production Ghost should later down-rank it for user-quality reasons. > > If: > > ```plaintext > qrels: > commercial -> low relevance > > and > > quality ranker: > commercial -> lower score > > ``` > > then part of the quality ranker’s evaluation advantage is built into the relevance definition. > > That is avoidable. > > A cleaner decomposition is: > > ```plaintext > retrieval question: > is this useful evidence for the stated information need? > > quality/policy question: > is this the kind of evidence Ghost should prefer in production? > > duplication question: > is this independent information or another copy? > > ``` > > Then you can report both relevance effectiveness and policy effectiveness without making them circular. > > For q18 specifically, your refined research-intent narrative may legitimately make many commercial “hire a hacker” pages irrelevant. I would just make that **topic-specific consequence of the information need** , rather than a universal commercial-page law. > **Ranking ablations: make each arm answer one question** > > The current provisional matrix is already useful because it tells you where the large deltas may be. > > Once the qrels stabilize, I would rerun it with maximally boring/atomic arms. > > Something like: > > ```plaintext > pure lexical BM25 > ↓ > BM25 + engine agreement > ↓ > BM25 + engine agreement + trust > > ``` > > and separately: > > ```plaintext > raw-rank RRF > ↓ > RRF + snippet-based ordering > ↓ > RRF + trust > ↓ > RRF + trust + quality > > ``` > > The point is not that Ghost should become a collection of isolated components in production. > > It is that an evaluation arm named: > > ```plaintext > BM25 > > ``` > > should ideally answer: > > > what does lexical BM25 contribute? > > and: > > ```plaintext > RRF > > ``` > > should answer: > > > what does rank fusion contribute? > > If the “plain” arm still contains another ordering heuristic, then the subsequent delta is harder to attribute. > > The final production configuration can still combine everything that wins. > > The experiment is simply easier to interpret. > > I would keep three outputs for each ablation: > > ```plaintext > mean metric > per-query delta > system-order impact > > ``` > > because the mean can hide a very asymmetric pattern such as: > > ```plaintext > trust helps 20 queries modestly > but catastrophically hurts 4 > > ``` > > or the reverse. > > That per-query heterogeneity is especially interesting with only 24 topics. > > * * * > > ### One evaluator/ranker interaction I would keep in mind > > If the same Nemotron model/family contributes both: > > ```plaintext > relevance judgments > > ``` > > and: > > ```plaintext > an LLM reranker being evaluated > > ``` > > I would add one independent control before treating a positive reranker delta as settled. > > [Balog, Metzler & Qin, “Rankers, Judges, and Assistants”](https://research.google/pubs/rankers-judges-and-assistants-towards-understanding-the-interplay-of-llms-in-information-retrieval-evaluation/) reports empirical evidence that LLM judges can be biased toward LLM-based rankers and can have difficulty resolving subtle ranking differences. > > That does not mean a Nemotron reranker improvement is fake. > > It means the strongest version of the result would be: > > ```plaintext > Nemotron reranker wins under all-five qrels > > and still wins under: > qrels excluding Nemotron > or > the small human-anchor subset > or > another independent judge variant > > ``` > > That is another cheap “remove the evaluator from the evaluated system” control. > **A few small reproducibility checks I would keep, but below the judge question** > > These are secondary to the qrels work, but they are cheap enough that I would keep them in the research artifact. > > ### 1. Clarify the 217/289 → 239/272 transformation > > One small accounting detail caught my eye. > > You report: > > ```plaintext > raw: 217 / 289 > corrected: 239 / 272 > > ``` > > and describe the corrected denominator as excluding 17 Nemotron infrastructure failures. > > Dropping 17 rows alone can reduce the denominator from 289 to 272, but it cannot increase the number of agreements from 217 to 239. > > So I assume “corrected” includes something additional such as: > > ```plaintext > rerunning failures > replacement judgments > post-parser-fix rejudging > > ``` > > If so, I would just record that transformation explicitly. > > Something like: > > ```plaintext > raw artifact > -> identify N infrastructure/parser failures > -> rerun/reparse according to rule X > -> corrected artifact > > ``` > > That will make the calibration table much easier for another researcher to reproduce. > > This is bookkeeping rather than a methodological problem. > > * * * > > ### 2. Keep parser/error provenance in the qrels artifact > > For each judge/item, I would retain: > > ```plaintext > raw model output > parsed grade > valid/invalid > retry count > error class > fallback used? # ideally always false in final qrels > model/backend/version > prompt/rubric version > > ``` > > Then the final aggregated qrel can always be reconstructed. > > A compact provenance manifest could hash: > > ```plaintext > frozen corpus > queries > narratives > judge prompt/rubric > pool > raw judge matrix > final qrels > > ``` > > You are already doing much of this, so I would continue in that direction. > > * * * > > ### 3. REST/stdio semantic parity and stdio purity are different tests > > Your 220/220 semantic-parity fixture is useful. > > I agree with your own note that **stdio purity** is a separate contract. > > The official [MCP stdio transport specification](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) reserves stdout for valid MCP messages; diagnostics belong on stderr. > > So the final cheap smoke test is something like: > > ```plaintext > spawn actual MCP subprocess > initialize > invoke ghost_search through a controlled/mock engine fixture > > assert: > every stdout frame is valid JSON-RPC/MCP > > allow: > diagnostics on stderr > > ``` > > That catches a class of failure that an in-process semantic-parity test cannot. > > I would not block qrels work on it. > > * * * > > ### 4. Tor configured-vs-observed is already the right separation > > I think your wording here is now basically right: > > ```plaintext > SafeSocks + per-search isolation credentials > = mechanism/configuration evidence > > ControlPort StreamID -> CircuitID observation > = runtime evidence that the actual requests were isolated > > ``` > > The latter is stronger if you eventually want an audit-grade runtime claim; it does not need to be part of the ranking benchmark. > > Likewise, if the 60–90 second timeout value is empirically chosen rather than traceable to a Tor Project recommendation, describing it as an empirical operational default is cleaner than forcing a citation. > **How I would interpret the five-judge question after all of those controls** > > For me, the decision tree would be approximately: > > ```plaintext > Does fixed-3 differ substantially from old-3? > | > yes > -> parser contamination was material > -> do not use old-3 vs fixed-5 to infer ensemble benefit > > Does fixed-5 improve human-anchor behavior over fixed-3? > | > yes > -> panel expansion has empirical value > | > no > -> five judges may still improve robustness, but not demonstrated by accuracy > > Do different 3-of-5 subsets / LOJO panels preserve retrieval-system order? > | > yes > -> panel composition is not a major benchmark fragility > | > no > -> identify which judge/examples produce the flips > > Do security-sensitive vs neutral paired prompts reveal judge-specific shifts? > | > yes > -> measure/mitigate that judge behavior directly > | > no > -> geographic balancing is probably solving a problem not visible in this benchmark > > Does a small human anchor reveal shared five-judge errors? > | > yes > -> calibrate rubric/prompt/panel against them > | > no > -> much stronger case for keeping automatic provisional qrels > > Do qrel variants preserve the same retrieval winner / pairwise ordering? > | > yes > -> benchmark is probably already useful for comparative IR work > | > no > -> understand the unstable topics before making ranker claims > > ``` > > So I would characterize your current reasoning as: > > > **sound motivation, reasonable ensemble experiment, but potentially over-specific if “3 Chinese / 2 Western” becomes the explanation for validity.** > > The strongest version of the story is instead: > > > Security-adjacent relevance assessment may have policy/refusal failure modes, so I deliberately used a heterogeneous judge panel; then I measured each judge’s failure behavior, compared panel subsets, anchored uncertain cases to a small human sample, and verified that the retrieval-system conclusions were stable. > > That is a much harder methodological story to attack than: > > > I chose a 3:2 geographic ratio because Western models might refuse security content. > > And importantly, it preserves the exact thing you are trying to protect: **a benchmark that does not silently grade down the security material its users actually need.** If I were minimizing work from your current state, my default order would now be: ```plaintext 1. Finish the fixed-parser five raw judge matrix. 2. Re-run the original same three judges with the fixed parser. This is the critical missing deconfound. 3. Offline: all 3-of-5 subsets all 4-of-5 LOJO panels fixed-3 vs fixed-5 4. Run a small paired security-sensitive/neutral wording control. 5. Human-label only the high-information sample: disagreements + boundaries + errors + security cases + random controls. 6. Compare: label agreement AND winner / pairwise retrieval-system ordering. 7. Finish judging the diverse four-arm pool and report judged@K. 8. Re-run the atomic ranking matrix against those stabilized qrels. 9. Only after that, move on to Nemotron query-refinement lift / neural reranking. ``` At this point I would not add a sixth judge, a more complicated voting scheme, or a heavier reranker just to increase apparent rigor. You already have enough machinery. The next gain is mostly from **separating the variables you already have**. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 30, 2026, 12:48am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/8 "2026-08-30T00:48:16Z")
Thanks for the clarification. I fully agree with the distinction: origin is a diversity heuristic, not the validity argument. The stronger claim is whether the five judges actually have complementary failure modes and produce more stable retrieval conclusions. That’s the frame I’ve used. I ran your 9-step sequence in the order you gave. Where things stand: **Steps 1–3 (parser/ensemble deconfound).** All three arms done: - A: original 3 judges + old parser - B: same 3 judges + fixed parser - C: all 5 judges + fixed parser The critical deconfound you flagged (B) is included. All three arms: `rrf-quality` wins, same pairwise order `RQ > RRF > BM25 > EC`. I then ran all 10 3-of-5 subsets and all 5 leave-one-judge-out panels offline — as you said, cheap once the 5-judge matrix exists. Across all 17 panels, winner and pairwise order are invariant. Document-level qrels do vary — up to 32.8% flip rate in subsets that exclude DeepSeek — but that variation doesn’t propagate to the system-level conclusion. **Step 4 (wording control).** You asked for this specifically, so the result here is the relevant one. Paired design: 6 queries, 30 stratified documents (1× grade 2, 2× grade 1, 2× grade 0 per query), 5 judges — 300 API calls, ~$0.217. Each document judged by the same judge under two framings: a security-sensitive information need (“I need this for investigating ransomware operations…”) and a neutral one (“I need this for general research about ransomware”). Document identity and judgment prompt were identical; only the framing differed. Per-judge deltas (security minus neutral, accuracy shares): | Judge | Security | Neutral | Delta | | --- | --- | --- | --- | | DeepSeek-R1 | 0.800 | 0.833 | -0.033 | | GPT-OSS-120B | 0.833 | 0.800 | +0.033 | | Nemotron-3-Super | 0.800 | 0.800 | 0.000 | | Llama-3.3-70B | 0.867 | 0.833 | -0.033 | | Kimi K2.6 | 0.867 | 0.700 | -0.167 | No systematic defensive refusal penalty on Western judges. Kimi is an outlier, but in the _opposite_ direction from the hypothesis — it grades _less_ strictly under security framing, not more. Cultural-group comparison: Chinese judges -0.067, Western judges -0.022, gap 0.044. The refusal mechanism Campbell et al. identified (2.72× refusal rate) is real, but it doesn’t manifest in judge grades on these queries and these documents. I’m not generalizing this to other models or other content domains — it’s a benchmark result, not a universality claim. **Step 5 (human anchor).** This is the one step I haven’t completed. I don’t have a human annotation pipeline. The closest substitute was validation against asukul’s RAG-Eval-LLM-Judge (NIST TREC RAG, 80-pair sample) — DeepSeek-R1 reached kappa=0.41 against human assessors, moderate, non-random. But that’s out-of-domain (TREC RAG isn’t onion security content), so it’s only a weak proxy. A small in-domain human anchor set over disagreements + boundary cases + security cases remains the highest-value missing validation. **Steps 6–8 (system-level comparison, judged@K, ranking rerun).** All done. judged@10: rrf=1.0, rrf-quality=0.892 (the quality filter pushes some unjudged docs into top-10). Atomic ablations reran on both qrels — winner and pairwise order stable across v2 and v3. **Step 9 (query refinement lift).** This was the last open step. I ran it with two separate arms, because you were right that acquisition and ranking are different questions: - **Frozen-candidate arm:** Same document set, ranked with raw vs. expanded query. BM25 nDCG@10 -0.0655 (term dilution hurts). RRF and RRF-quality are controls here — both +0.0000, since candidates are identical and RRF doesn’t use query terms. - **Live-acquisition arm:** 24 query pairs over 13 live onion search engines via Tor, 624 requests, ~30 minutes. Expanded queries retrieve 4.26x fewer documents (mean 1110 vs 260) with 6% pool overlap (Jaccard). All four strategies show negative lift: | Strategy | Raw nDCG@10 | Expanded nDCG@10 | Lift | | --- | --- | --- | --- | | engine-count | 0.2660 | 0.0500 | -0.2160 | | BM25 | 0.2915 | 0.0615 | -0.2300 | | RRF | 0.3474 | 0.0242 | -0.3232 | | RRF-quality | 0.4324 | 0.0601 | -0.3723 | The mechanism is twofold: (1) onion search engines treat long queries as AND/exact-phrase and return fewer results; ahmia errors on 10/24 expanded queries, tordex on 19/24. (2) The different documents they do return have only 6% overlap with the raw pool, so most qrels-annotated relevant documents are lost. Both arms agree: no lift. Production config keeps raw queries. **What I don’t claim.** The qrels remain LLM-generated and provisional. I haven’t human-reviewed any of the 1197 pairs. Panel robustness (17 panels, invariant conclusion) and NIST validation (moderate agreement, non-random) are what I have — they’re robustness and non-randomness evidence, not correctness evidence. A small in-domain human anchor set (step 5) is the validation that would strengthen this most, and I don’t currently have a way to do it. **What I took from your advice.** The “measure judges as instruments, not by origin” framing shaped the wording control design — test whether the failure mechanism shows up in actual outputs rather than extrapolating from origin. The deconfound step ordering (old-3 → fixed-3 → fixed-5) was useful: the end-to-end old-3 vs fixed-5 comparison is informative as a before/after, but the internal comparisons (old-3 vs fixed-3 for parser, fixed-3 vs fixed-5 for ensemble) tell me the parser fix was the larger effect and adding GLM/Kimi didn’t change the conclusion. Separating variables rather than adding judges was the right call. I didn’t add a sixth judge, a heavier reranker, or a more complex voting scheme. The conclusion is stable within the 5 judges I have. The next phase is applying the findings to the production system, not more benchmark iteration. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [August 30, 2026, 3:49am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/9 "2026-08-30T03:49:04Z")
Step 5 is done. Here’s the result. **Human anchor annotation.** 288 items, stratified per your specification: | Stratum | Count | Purpose | | --- | --- | --- | | 3-2 splits | 207 | Panel genuinely divided — human label is decisive | | Boundary cases (final=1 with dissent) | 50 | Grade threshold unclear | | Unanimous grade 2 | 11 | Positive control | | Unanimous grade 0 | 20 | Negative control | Single annotator — domain expert with onion/security research experience. 62 documents had no snippet and required Tor page fetch to assess. I built a local annotation tool with auto-save and resume so the work could be done iteratively across sessions. **Cohen’s kappa (human vs each LLM judge):** | Judge | Kappa | Interpretation | Agreement | | --- | --- | --- | --- | | DeepSeek-R1 | 0.842 | almost perfect | 89.6% | | GPT-OSS-120B | 0.340 | fair | 55.6% | | Llama-3.3-70B | 0.336 | fair | 55.4% | | Nemotron-3-Super | 0.117 | slight | 38.5% | | Kimi K2.6 | 0.074 | slight | 35.1% | | LLM Majority | 0.454 | moderate | 62.5% | DeepSeek-R1 is the only judge with “almost perfect” human agreement. Kimi (0.074) and Nemotron (0.117) are unreliable as standalone judges — Kimi due to the `max_tokens=512` truncation artifact I documented earlier, Nemotron for reasons I haven’t investigated yet. The majority vote (0.454, moderate) is better than any individual judge except DeepSeek, but still disagrees with the human on 37.5% of cases. **Fleiss’ kappa (human + 5 LLM judges together):** 0.389 (fair). **Controls confirmed.** Unanimous grade 0: 100% agreement (20/20). Unanimous grade 2: 100% agreement (11/11). The human agrees with the LLM panel on clear cases — the annotation task was calibrated correctly. **Per-stratum agreement (human vs majority):** - 3-2 splits: 60.4% — the human disagrees with the LLM majority on 40% of genuinely disputed cases. - Boundary cases: 48.0% — the human disagrees on half, which is expected for grade-threshold-ambiguous documents. **Per-category agreement:** - threat\_intel: 56.1% — lowest, security content is genuinely harder to judge. - journalism: 67.2%. - privacy: 66.4%. **System-order rerun.** I replaced 95 majority labels with human labels in qrels\_v3 (where the human disagreed) and reran all 4 ranking strategies on the frozen SERP corpus: | Strategy | Original nDCG@10 | Human-corrected nDCG@10 | Delta | | --- | --- | --- | --- | | engine-count | 0.2762 | 0.2776 | +0.0014 | | BM25 | 0.2915 | 0.2811 | -0.0105 | | RRF | 0.3592 | 0.3478 | -0.0114 | | RRF-quality | 0.4594 | 0.4080 | -0.0515 | Absolute nDCG values drop slightly — the human labels are somewhat stricter than the LLM labels. But the system ordering is preserved: - Original: `rrf-quality > rrf > bm25 > engine-count` - Human-corrected: `rrf-quality > rrf > bm25 > engine-count` **The ranking conclusion survives human correction of 95 disputed labels.** **What this validates and what it doesn’t.** This validates that the system-level ranking conclusion is robust to human correction of the most contested labels. It does not validate that every individual qrel label is correct — the human disagreed with the LLM majority on 108 of 288 items (37.5%), and 95 of those changed labels. What it shows is that label-level disagreement doesn’t propagate to the system-level conclusion, which is the question you asked in step 6. The single-annotator design is a limitation — there’s no human-human kappa, only human-vs-LLM kappa. A second annotator would strengthen this. But the controls (100% on unanimous cases) and the DeepSeek kappa (0.842) suggest the annotation is reliable. **Where the full sequence stands now:** All 9 steps are done. The ranking conclusion (`rrf-quality > rrf > bm25 > engine-count`) is stable across: - 17 panel compositions (A/B/C deconfound + 10 3-of-5 subsets + 5 LOJO) - Wording control (no defensive refusal bias) - Human anchor (95 corrected labels, order preserved) - Query refinement (no lift in either arm) The benchmark cycle is closed. The next phase is applying the findings to the production system. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [August 30, 2026, 7:35am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/10 "2026-08-30T07:35:01Z")
When I replied last time, I could only see one of the posts… 😅 And somehow the repo seems to have disappeared again!? 😂 But anyway, I think we can probably call the benchmark complete: * * * At least for the question you originally needed the benchmark to answer — > **which ranking configuration should Ghost use in production, and is that choice robust to reasonable changes in the qrels/judges?** —I think you now have enough evidence to close this benchmark cycle and move on. The important result is no longer just that `rrf-quality` has the highest mean nDCG. It is that the same production conclusion survived several deliberately different perturbations: ```plaintext old 3-judge + contaminated parser ↓ same 3 judges + fixed parser ↓ 5 judges + fixed parser ↓ all 3-of-5 subsets ↓ all leave-one-judge-out panels ↓ security-sensitive / neutral wording control ↓ 288-item human stress sample ↓ 95 qrel corrections ``` and the winner remained: ```plaintext rrf-quality > rrf > bm25 > engine-count ``` That is much more informative than simply accumulating another judge or another reranker. In particular, the human-anchor result closes the loop I was most interested in: **the individual labels can be quite noisy while the system-level conclusion remains stable.** That distinction is well established in IR evaluation. TREC work has long shown that assessor disagreement need not translate into the same degree of instability in comparative system evaluation; the practical question is whether the retrieval-system conclusions survive reasonable judgment perturbations, not whether every relevance label is universally agreed upon. See, for example, [Voorhees, “Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness”](https://www.nist.gov/publications/variations-relevance-judgments-and-measurement-retrieval-effectiveness). So my default path from here would actually be: ```plaintext freeze benchmark v1 ↓ freeze production configuration ↓ use benchmark v1 as a regression suite ↓ move development effort back to production/search behavior ``` rather than continuing to enlarge the current benchmark until it becomes its own project. I would keep only a few **publication/reusability hardening** items separate from the production decision. None of them looks like a reason to hold production work. | Question | My current read | | --- | --- | | Which production ranker? | **Answered:** `rrf-quality` | | Does parser contamination explain the winner? | **No:** fixed-parser comparison preserves it | | Does one particular 5-judge composition determine the winner? | **Apparently no:** subset/LOJO stability | | Did the hypothesized Western-judge grade penalty appear here? | **Not in the paired grade-control result** | | Does human correction overturn the winner? | **No** | | Should query expansion be enabled now? | **No; raw queries are the safer production choice** | | Is every qrel now “validated ground truth”? | **No, and it does not need to be for the production decision** | | Is the collection publication-ready without any caveats? | **Almost; a few cheap documentation/coverage closures remain** | > **Why I think the benchmark cycle can close** > > ### 1. The deconfound worked > > The most important methodological improvement since the previous post is that you actually ran the missing middle condition: > > ```plaintext > A = old 3 judges + old parser > B = same 3 judges + fixed parser > C = 5 judges + fixed parser > > ``` > > So the two effects are no longer silently mixed: > > ```plaintext > A -> B > parser/error-semantics effect > > B -> C > judge-panel/composition effect > > ``` > > That was the main thing missing from the earlier `old-3 vs fixed-5` comparison. > > The fact that the system winner survives A/B/C means you no longer need to infer robustness from an end-to-end before/after comparison. > > * * * > > ### 2. The panel-composition question is basically answered empirically > > The original concern was: > > > could security-adjacent material be systematically downgraded because some judges have different refusal/safety behavior? > > The right response was to measure the judges rather than infer validity from model origin, and the new experiments do that. > > You now have: > > ```plaintext > 10 × 3-of-5 subsets > 5 × leave-one-judge-out panels > paired security/neutral framing > per-judge agreement behavior > human anchor > > ``` > > and the production ranking survives the panel perturbations. > > That is a stronger result than saying: > > ```plaintext > 3 Chinese-origin + 2 Western-origin > > ``` > > is intrinsically better. > > In fact, the paired wording test is useful specifically because the expected Western-only grade penalty did **not** appear. > > I would phrase that result narrowly: > > > On these queries/documents, this paired control did not show a systematic security-framing grade penalty concentrated in the Western judges. > > rather than: > > > defensive refusal bias does not exist. > > The latter would go beyond the experiment. Campbell et al.'s [Defensive Refusal Bias](https://arxiv.org/abs/2603.01246) result is about refusal behavior under semantically equivalent security-sensitive vs neutral prompts; your experiment is a benchmark-specific paired relevance-judgment control. The two fit together nicely precisely because yours asks whether that known mechanism is actually visible in **this** evaluation pipeline. > > And apparently, at least in the public results you reported, it is not driving the ranking conclusion. > > That means the origin-composition question can probably be retired unless a future model change reopens it. > > * * * > > ### 3. Document-level instability with system-level stability is not a contradiction > > The human result is especially informative here. > > On the deliberately difficult 288-item sample: > > ```plaintext > human vs LLM-majority agreement = 62.5% > 108 / 288 disagreements > 95 qrel labels changed > > ``` > > That sounds alarming if the target is: > > > every LLM qrel should reproduce a human judgment. > > But then the ranking rerun gives: > > ```plaintext > original: > rrf-quality > rrf > bm25 > engine-count > > human-corrected: > rrf-quality > rrf > bm25 > engine-count > > ``` > > So the benchmark is telling you something subtler: > > ```plaintext > label-level validity: > still imperfect > > comparative system-level validity: > substantially more stable > > ``` > > For a benchmark whose immediate purpose is choosing among retrieval systems, the latter is the property that matters most. > > This is why I would not keep adding automatic judges just to raise the apparent agreement number. > > * * * > > ### 4. I would make the current corpus immutable now > > At this point I would version the current state as something like: > > ```plaintext > ghost-serp-benchmark-v1 > > frozen candidate corpus > frozen 24 topics/narratives > frozen qrels > frozen human corrections > frozen ranker parameters > frozen evaluation implementation > > ``` > > and stop changing it in-place. > > Then production work can use it as a regression suite: > > ```plaintext > change production code > ↓ > replay benchmark v1 > ↓ > did effectiveness regress? > did candidate identity change unexpectedly? > did judged@K collapse? > > ``` > > If you later want stronger scientific generalization, create: > > ```plaintext > benchmark v2 > > ``` > > from new queries and/or a fresh acquisition snapshot. > > That is cleaner than continuing to tune and re-label v1 forever. > **The human anchor is strong evidence, but I would keep two interpretation boundaries explicit** > > ### 1. It is a stress sample, not a random estimate of 1,197-item accuracy > > Your sampling deliberately emphasizes: > > ```plaintext > 207 × 3-2 splits > 50 × boundary cases > 11 × unanimous grade 2 > 20 × unanimous grade 0 > > ``` > > That is exactly the kind of sample I would want for **stress-testing the system conclusion**. > > But because disputed cases are intentionally overrepresented, numbers such as: > > ```plaintext > DeepSeek kappa = 0.842 > LLM-majority kappa = 0.454 > > ``` > > should not automatically be interpreted as estimates of what those kappas would be over all 1,197 documents. > > I would describe them as: > > > agreement on a stratified, disagreement-heavy human audit sample. > > That is actually a useful strength, not a weakness: you pointed human effort at the cases most likely to change conclusions. > > * * * > > ### 2. The 62 page-fetched items are not exactly the same relevance task > > This is worth preserving in the benchmark metadata. > > You said 62 items lacked enough snippet evidence and the human assessor fetched the Tor page. > > If the LLM judges saw only: > > ```plaintext > URL > title > snippet > > ``` > > while the human saw: > > ```plaintext > URL > title > snippet > full page > > ``` > > then those 62 comparisons contain two changes at once: > > ```plaintext > different assessor > + > more evidence > > ``` > > There is prior IR work showing that full text can change human relevance judgments relative to URL/title/snippet summaries. [Kazai et al., “Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?”](https://www.microsoft.com/en-us/research/publication/less-is-less-when-are-snippets-insufficient-for-human-vs-machine-relevance-estimation/) studies exactly this distinction. > > So if the underlying rows are still available, one essentially free analysis would be: > > ```plaintext > 226 surrogate-only human judgments > vs > 62 page-assisted human judgments > > ``` > > and report agreement separately. > > No new annotation is required. > > If the page-assisted subset has a much higher LLM/human disagreement rate, that may simply tell you: > > > some SERP surrogates do not contain enough evidence. > > That is a useful broker-level measurement in its own right. > > It could eventually justify: > > ```plaintext > judgment_status = insufficient_surrogate_evidence > > ``` > > instead of forcing every candidate into `0/1/2`. > > * * * > > ### 3. One human assessor is enough for this stress test, but not for a strong human-ground-truth claim > > The unanimous controls are reassuring sanity checks: > > ```plaintext > unanimous grade 0 -> 20/20 human agreement > unanimous grade 2 -> 11/11 human agreement > > ``` > > and the DeepSeek/human agreement is interesting. > > But neither independently measures **human assessor reliability**. > > For that, you would need a second human on at least some overlapping items. > > I would therefore distinguish: > > ```plaintext > production benchmark closure: > single expert stress audit is already useful > > publication-quality human ground truth: > second independent human subset would strengthen it > > ``` > > I would not delay production work for the second one. > **One nuance about the final ranking order** > > I agree that the **winner** is robust. > > I would be a little more careful saying that every pairwise difference is equally well established. > > After human correction: > > ```plaintext > rrf-quality 0.4080 > rrf 0.3478 > bm25 0.2811 > engine-count 0.2776 > > ``` > > The `rrf-quality` margin remains fairly substantial. > > But: > > ```plaintext > BM25 - engine-count = 0.0035 nDCG@10 > > ``` > > is tiny. > > With only 24 topics, small mean differences can be quite sensitive to which topics happen to be in the evaluation set. The classic NIST study [The Effect of Topic Set Size on Retrieval Experiment Error](https://www.nist.gov/publications/effect-topic-set-size-retrieval-experiment-error) specifically shows that with small topic sets, apparently small system differences can reverse under another sample of topics. > > So I would make the strongest claim: > > > `rrf-quality` is the robust production winner among the evaluated configurations. > > and a weaker secondary statement: > > > the observed complete order was `RQ > RRF > BM25 > EC`. > > rather than making the 0.0035 lower-pair margin carry the same evidentiary weight as the winner result. > > If the per-topic scores already exist, a paired bootstrap/randomization interval for those lower comparisons is cheap; but again, I would consider that publication polish, not a reason to reopen the production decision. > **The remaining qrels-coverage issue is small enough to close cheaply** > > You now report: > > ```plaintext > rrf judged@10 = 1.000 > rrf-quality judged@10 = 0.892 > > ``` > > For 24 queries × 10 positions, `0.892` corresponds roughly to only a few dozen top-rank slots being unjudged. > > That is not enough for me to discard the result, especially because the human-corrected rerun still preserves the winner. > > But if you want to turn this into a reusable public benchmark, this is probably the cheapest remaining qrels improvement: > > ```plaintext > take union of current systems' Top-10 > or Top-20 > ↓ > judge every still-unjudged candidate > ↓ > freeze qrels > > ``` > > No need to re-annotate the whole 28k corpus. > > The reason is the standard incomplete-pooling problem: evaluation sets created through pooling can disadvantage systems that retrieve documents that were not contributed to the judged pool. NIST discusses this directly in [Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements](https://www.nist.gov/publications/reliable-information-retrieval-evaluation-incomplete-and-biased-judgements). > > For current production selection, I would call this a caveat. > > For a public benchmark that future neural rankers may retrieve against, I would close it. > > I would also preserve: > > ```plaintext > judged@10 > judged@20 > > ``` > > for every future system. > > Then a future model with a low nDCG score but 40% unjudged Top-20 is clearly distinguishable from a genuinely poor ranker. > **Query refinement: I think the production decision is answered, but I would phrase the mechanism carefully** > > I agree with the production decision: > > ```plaintext > keep raw queries > > ``` > > The experiment gives two independent reasons. > > ### Frozen candidate arm > > For BM25: > > ```plaintext > expanded query -> nDCG@10 -0.0655 > > ``` > > On the same documents, that is fairly direct evidence that the expansion diluted the lexical representation used by BM25. > > The RRF controls correctly stay unchanged because they do not depend on the query text when candidate lists/ranks are fixed. > > That is a nice clean experiment. > > * * * > > ### Live acquisition arm > > The operational result is even more striking: > > ```plaintext > raw mean candidates ~1110 > expanded mean candidates ~260 > > ~4.26× candidate loss > > ``` > > plus substantial engine errors and only about 6% overlap between raw and expanded pools. > > That is more than enough to say: > > > this expansion scheme is not suitable for the current production metasearch path. > > I would only separate that from the stronger statement: > > > the expanded-only documents themselves are less relevant. > > Because the expanded pool is almost completely different, many expanded-only documents were presumably outside the original judgment pool. > > Under ordinary pooled evaluation, unjudged documents can be treated as nonrelevant, so the live-arm nDCG can punish a system simply for retrieving unseen material. That is the same incomplete-judgment issue discussed above. > > So I would interpret the live result as two layers: > > ```plaintext > very strong: > expansion hurts candidate yield / engine compatibility > > not yet isolated: > expanded-only documents are intrinsically less relevant > > ``` > > For the production decision, the first is enough. > > * * * > > ### The engine failure mechanism is also worth wording more generally > > I would not summarize the failure as: > > > onion engines treat long queries as AND/exact phrase. > > At least Ahmia’s current public implementation is more specific. > > Its search view rejects queries over **100 characters or more than 10 space-separated terms** , and its Elasticsearch query uses `minimum_should_match: "75%"`, not a literal exact-phrase requirement. The current implementation is visible in [Ahmia’s `views.py`](https://github.com/ahmia/ahmia-site/blob/master/ahmia/views.py). > > So the cleaner generalization is: > > > **free-form LLM expansion collided with heterogeneous engine query contracts.** > > Different upstream engines may have: > > ```plaintext > length limits > term-count limits > tokenization differences > AND-like behavior > minimum-match thresholds > form/query syntax > different ranking semantics > > ``` > > That is a more interesting federated-search result anyway. > > The Ghost broker cannot assume that one verbose reformulation is a valid query language for every remote engine. > > * * * > > ### If query refinement ever comes back, I would change the formulation of the problem > > Instead of: > > ```plaintext > raw query > ↓ > LLM writes one richer query > ↓ > send same expansion to all engines > > ``` > > I would test something closer to: > > ```plaintext > information need > ↓ > query planner > ↓ > short atomic queries / engine-compatible reformulations > ↓ > upstream engines > ↓ > fusion > > ``` > > For example: > > ```plaintext > "recent ransomware payment infrastructure and laundering methods" > > -> ransomware payments > -> ransomware bitcoin wallets > -> ransomware laundering > > ``` > > rather than a sentence-length paraphrase. > > That becomes a **query planning / federated acquisition** experiment, not just “does an LLM rewrite look better?” > > But because the present expansion has clearly failed the production test, I would put this in a future branch and move on. > **RRF k=10: fine for production, but I would treat v1 as the development collection** > > The switch from `k=60` to `k=10` seems perfectly reasonable as an empirical production choice. > > One useful historical detail is that the original [Cormack, Clarke & Büttcher RRF paper](https://cormack.uwaterloo.ca/cormack/cormacksigir09-rrf.pdf) also used a pilot to choose the fusion constant and then kept it fixed for subsequent validation. > > That suggests a clean boundary for Ghost: > > ```plaintext > 24-query frozen corpus > = development / configuration-selection collection > > selected: > RRF k=10 > quality configuration > trust weighting > > ``` > > Then freeze them. > > If later you want to claim: > > > k=10 is generally preferable for onion metasearch > > rather than: > > > k=10 works best for the current Ghost benchmark > > the next test should be a **new query set or later frozen SERP snapshot** , with `k=10` fixed before looking at the new qrels. > > That is not unfinished work for benchmark v1. > > It is benchmark v2 / external validation. > **A few artifact/provenance details I would clean up before publishing the benchmark** > > These are mostly documentation issues now. > > ### 1. I would call qrels\_v2 and qrels\_v3 “qrel variants” rather than “independent qrels” > > They use the same topic set and judgment pool, and share several judges. > > So: > > ```plaintext > two qrel variants > two judgment conditions > two ensemble/parser configurations > > ``` > > is more precise than: > > ```plaintext > two independent qrels sets > > ``` > > The interesting result does not depend on independence anyway. > > What matters is: > > > materially different judgment conditions still produced the same winner. > > * * * > > ### 2. Make the panel-count convention explicit > > You describe: > > ```plaintext > A / B / C > 10 × 3-of-5 > 5 × LOJO > > ``` > > and elsewhere refer to 17 panels. > > On paper those labels can look like 18 named conditions unless one of the categories overlaps another condition. > > There may be a perfectly simple counting convention behind this; I would just state it in the artifact so a future reader does not spend time reverse-engineering the number. > > * * * > > ### 3. Pin exact model/provider IDs > > There is some naming drift across the thread: > > ```plaintext > DeepSeek V4 Flash > DeepSeek-R1 > > Kimi-K3 > Kimi K2.6 > > ``` > > and GLM was replaced with Llama before the canonical five-judge run. > > That is all manageable during exploratory work, but the final benchmark manifest should make each judgment reproducible: > > ```plaintext > judge_id > provider > exact provider model string > model family/version > date > prompt hash > max_tokens > temperature > parser version > retry policy > > ``` > > This matters especially because one of the interesting findings is itself a model-specific failure mode: Kimi’s grade distribution, Nemotron’s 429s, GLM parse behavior, etc. > > * * * > > ### 4. Keep relevance labels separate from the production quality policy > > The quality stage includes things such as: > > ```plaintext > domain collapse > near-duplicate demotion > information-quality scoring > commercial-intent penalty > mirror detection > > ``` > > while the relevance narratives also encode a research intent. > > That is reasonable for production, but for a reusable benchmark I would preserve enough structure that a future researcher can distinguish: > > ```plaintext > topical relevance > quality/policy preference > duplication > commercial intent > > ``` > > rather than baking them irreversibly into one label. > > This is not an argument that `rrf-quality` should lose. > > The human-corrected result suggests the production configuration is robust. > > It is just an artifact-design choice that will make the benchmark easier to reuse for a different downstream policy. > **About the external NIST sanity check** > > I would keep the TREC RAG comparison, but as exactly what you now call it: an **out-of-domain sanity check**. > > It establishes something useful: > > ```plaintext > the judges are not producing arbitrary/random relevance behavior > > ``` > > against an external human-labeled collection. > > What I would avoid is assigning the performance gap to: > > ```plaintext > domain mismatch > scale collapse > ensemble size > > ``` > > as though their individual magnitudes were already measured. > > All three are plausible explanations, but separating them would require controlled experiments. > > Fortunately, you do not need that causal decomposition. > > The in-domain human stress audit is now much more relevant to the Ghost benchmark itself. > > So the external NIST result can stay in the supporting-evidence layer: > > ```plaintext > external out-of-domain sanity check > + > in-domain human stress correction > + > system-order stability > > ``` > > Together they tell a stronger story than any one of them alone. So I think the project has reached a useful transition point: ```plaintext BENCHMARK V1 ============ Acquisition/frozen corpus done Topic narratives done Diverse judgment pool done Parser deconfound done Judge-panel perturbation done Wording sensitivity control done External human-qrel sanity done In-domain human stress audit done Atomic ranking comparison done System-order robustness done judged@K accounting done query-refinement production test done Production decision: rrf-quality raw queries ``` and then a separate optional layer: ```plaintext PUBLICATION / BENCHMARK HARDENING ================================= nice to have: - fill remaining top-rank unjudged candidates - split surrogate-only vs page-assisted human results - exact model/run provenance - second human subset if stronger human-ground-truth claims are desired - held-out/fresh benchmark v2 for external generalization ``` I would not let that second list turn back into a prerequisite for shipping the production findings. The useful shift now is from: > “How many more ways can I validate this benchmark?” to: > **“How do I freeze what I learned and make sure future production changes do not regress it?”** That gives the benchmark a continuing role without making benchmark construction the permanent center of the project. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [September 2, 2026, 12:34pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/11 "2026-09-02T12:34:09Z")
Thanks for the comprehensive review. The repo is back public on Codeberg — I had briefly set it to private during a refactor, it’s visible again at the link in the original post. I agree the benchmark cycle can close. Here’s where each point landed. ## Benchmark v1 frozen Frozen as `ghost-serp-benchmark-v1`: - Frozen candidate corpus (24 queries, 13 engines, ~28,700 results) - Frozen 24 topics/narratives - Frozen qrels (v2 + v3 as variants, see below) - Frozen human corrections (95 labels, order preserved) - Frozen ranker parameters (RRF k=10, quality config, trust weighting) - Frozen evaluation implementation Production configuration: `rrf-quality`, raw queries, RRF k=10. The benchmark now serves as a regression suite: change production code → replay v1 → check for regressions. ## Surrogate-only vs page-assisted split — done You suggested this as an essentially free analysis. It was — the data was already there, I just hadn’t split it. Results: | Subset | n | Agreement | Cohen’s kappa | Interpretation | | --- | --- | --- | --- | --- | | Surrogate-only | 226 | 66.4% | 0.507 | moderate | | Page-assisted | 62 | 48.4% | 0.272 | fair | | Delta | | +18.0pp | +0.235 | | The page-assisted subset has substantially lower LLM/human agreement. This is consistent with your reference to Kazai et al. — the human saw full page text while the LLM judges saw only URL/title/snippet. The two comparisons contain two changes at once (different assessor + more evidence), so I’m not claiming the gap is purely about evidence sufficiency. But the direction is clear: some SERP surrogates do not contain enough information for LLM judges to make accurate relevance judgments. The page-assisted items are concentrated in `threat_intel` (55 of 62), which is already the hardest category (surrogate-only threat\_intel agreement: 62.7% vs journalism 69.0% vs privacy 67.0%). So the gap is partly confounded with category difficulty. But even within threat\_intel, the surrogate-only agreement (62.7%) is higher than the page-assisted agreement (49.1%). This supports your suggestion of a `judgment_status = insufficient_surrogate_evidence` field rather than forcing every candidate into 0/1/2. I’ll add this to the benchmark metadata for v1 — not as a new label, but as a flag on the 62 items where the human needed page fetch. Per-category breakdown: | Category | Surrogate agree | Page-assisted agree | | --- | --- | --- | | threat\_intel | 37/59 (62.7%) | 27/55 (49.1%) | | journalism | 40/58 (69.0%) | 3/6 (50.0%) | | privacy | 73/109 (67.0%) | 0/1 (0.0%) | The privacy page-assisted count is too small (n=1) to interpret. Threat\_intel and journalism both show the expected pattern. ## Ahmia correction — accepted You’re right. I oversimplified the engine failure mechanism. Ahmia’s `views.py` rejects queries over 100 characters or more than 10 space-separated terms, and uses `minimum_should_match: "75%"` — not a literal exact-phrase requirement. The cleaner generalization is: > Free-form LLM expansion collided with heterogeneous engine query contracts. Different engines have length limits, term-count limits, tokenization differences, minimum-match thresholds, and different query syntax. The Ghost broker cannot assume one verbose reformulation is a valid query language for every upstream engine. I’ve corrected this in my internal documentation. ## Pairwise margin caveat — accepted The strongest claim is: > `rrf-quality` is the robust production winner among the evaluated configurations. The weaker secondary statement: > The observed complete order was `RQ > RRF > BM25 > EC`. I won’t make the BM25 vs engine-count margin (0.0035 nDCG@10 after human correction) carry the same evidentiary weight as the winner result. With 24 topics, that margin is within the noise range that Voorhees and the NIST topic-set-size literature warn about. If I later want to claim the full ordering is robust, I’d need a paired bootstrap or a larger topic set. For now, the production decision only depends on the winner. ## qrels naming — accepted I’ll refer to qrels\_v2 and qrels\_v3 as “qrel variants” rather than “independent qrels” going forward. They share the same topic set, judgment pool, and several judges. The interesting result — materially different judgment conditions producing the same winner — does not depend on independence. ## Model/provider ID pinning The provenance files already capture: model string, endpoint (OpenRouter), avg latency, error count, prompt version, prompt hash, pool version, narrative version, decision method, and refinements applied (including the GLM→Llama swap and the Kimi max\_tokens issue). Missing from the current provenance: `max_tokens`, `temperature`, `parser version`. These will be added to the frozen v1 manifest. For reference: - `max_tokens`: 512 for all judges in v2/v3 (this was the root cause of Kimi’s truncation — documented separately) - `temperature`: 0 for all judges - Parser: JSON codeblock strip + keyword fallback, version `v2-calibrated-ensemble` The Kimi `max_tokens=512` artifact is documented in the provenance refinements: “Kimi K2.6 is a reasoning model that produces chain-of-thought before JSON output. At 512 tokens, the JSON is frequently truncated, triggering parse fallback.” The head-to-head test (max\_tokens=4096, 25 docs) showed Kimi at 84% exact match vs 88% for Qwen3-Max — the model itself was fine, the configuration was wrong. This is a model-specific failure mode that the provenance captures. ## Relevance vs quality policy separation Acknowledged as an artifact-design choice for v2. The current `rrf-quality` configuration bakes together: - Topical relevance (the qrels labels) - Quality/policy preferences (domain collapse, near-duplicate demotion, information-quality scoring, commercial-intent penalty, mirror detection) For v1 as a production regression suite, this is fine — the production system uses the combined configuration. For a reusable public benchmark, I’ll preserve enough structure that a future researcher can separate topical relevance from quality/policy. Not blocking v1 freeze. ## Query planner — agreed If query refinement comes back, the formulation should be: ```auto information need → query planner → short atomic queries / engine-compatible reformulations → upstream engines → fusion ``` not “LLM writes one richer query → send to all engines.” The federated-search framing is the right one — the broker needs to speak each engine’s query language, not force one verbose reformulation through every upstream. This goes in a future branch, not v1. ## External NIST sanity check Keeping it as exactly what it is: an out-of-domain sanity check that establishes the judges are not producing arbitrary/random relevance behavior. Not assigning the performance gap to individual causes (domain mismatch, scale collapse, ensemble size) — those would need controlled experiments. The in-domain human stress audit is the relevant validation for the Ghost benchmark itself. ## What’s next ```auto BENCHMARK V1 — frozen, serves as regression suite PRODUCTION — move development effort back to search behavior: - engine health monitoring (13 engines, heterogeneous failure modes) - engine-specific query contracts (length limits, term limits, match thresholds) - snippet quality detection (flag insufficient surrogates before judging) - production ranking: rrf-quality, raw queries, RRF k=10 PUBLICATION HARDENING (optional, not blocking): - fill remaining top-rank unjudged candidates (rrf-quality judged@10 = 0.892) - second human annotator on a subset for human-human kappa - benchmark v2 from fresh queries for external generalization ``` The useful shift you described — from “how many more ways can I validate this benchmark?” to “how do I freeze what I learned and make sure future production changes do not regress it?” — is the one I’m taking. Thanks for the methodology guidance throughout. The “measure judges as instruments, not by origin” framing, the deconfound step ordering, and the federated-search problem setting all shaped the experiment design in ways that made the results stronger than they would have been otherwise. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 3, 2026, 12:20am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/12 "2026-09-03T00:20:38Z")
I think you’re good to move into production.😀 I went looking for the landmines I could see: * * * I would keep benchmark v1 frozen at this point. It has already done the job you needed for the production decision: it selected `rrf-quality`, raw queries, and RRF `k=10`, and the later judgment stress tests did not give a strong reason to reopen that decision. The one thing I would pin before the refactor gets much farther is the boundary between **the frozen benchmark implementation** and **the ranker that production actually executes**. My default route would be: ```plaintext frozen v1 evidence ├── frozen reference/evaluator │ corpus + qrels + metric definitions + historical baseline │ └── current production ranker system under test ``` In other words: I would **not** rewrite the frozen evaluator to follow production. Keep it frozen as the reference. Instead, add a thin adapter that feeds the same frozen SERPs into the _actual current production composition_, then evaluates that output against the frozen v1 qrels. That makes the intended workflow literal: ```plaintext change production code ↓ run the current production ranker on frozen v1 inputs ↓ compare with frozen v1 evidence ↓ either: behavior preserved or: behavior intentionally changed, with measured quality impact ``` That seems higher-value to me than another benchmark round. I would also give the selected production composition a named profile — something like `production-v1` — rather than relying on a collection of defaults. I do **not** mean changing Ghost’s global defaults; the off-by-default filters have their own policy/liability rationale. I mean making the production deployment’s intended composition explicit and executable. Roughly: ```plaintext production-v1 query policy = raw RRF = enabled RRF k = 10 quality filter = enabled RRF input ordering = pinned QF candidate depth = pinned QF feature set = pinned duplicate representative = deterministic snippet/ranking policy = explicit dedup policy = explicit output limit semantics = explicit profile/version id = production-v1 ``` The reason I would pin the _composition_, not only `rrf-quality`, is that there are currently a few reachable ways for two paths both called “RRF + quality filter” to produce different rankings. I tried a small offline synthetic fixture against those seams. It is **not** evidence that the private frozen benchmark loses nDCG/MAP/MRR; I cannot measure that without the frozen corpus/qrels. What it does show is the narrower thing that matters here: some of the current benchmark/runtime paths are not semantically equivalent in general. The two cheapest production fixes I would probably do immediately are even smaller: 1. **restore/enforce stdio purity** — normal preflight/debug logging should never reach the MCP stdout stream; 2. **do not let `HTTP 200 + zero parsed results` automatically collapse into “healthy empty result”** — preserve enough acquisition state to distinguish genuine empty SERPs from challenge/interstitial/parser failure. Those are both small changes with unusually high diagnostic value. > **Why I would make production itself the regression SUT** > > ### 1. `rrf-quality` is a pipeline, not just two switches > > The interesting drift is not only whether `qualityFilter` is `true` or whether the RRF constant is `10`. > > There are several places where ranking semantics can change before those headline settings even matter. > > #### A. Per-engine ordering before RRF > > RRF consumes **ranks** , not the underlying relevance scores. So anything that changes the per-engine order before RRF changes the fusion input. > > The frozen evaluation path and the current MCP path do not appear to construct those per-engine ranks identically: one path can reorder using snippet informativeness before assigning RRF ranks, while the runtime path can inherit parser-return order. > > That means: > > ```plaintext > same documents > same RRF formula > same k=10 > different input ranks > ↓ > possibly different fused ranking > > ``` > > This is why I would treat “per-engine normalization/order before fusion” as part of `production-v1`. > > It is also a general RRF property: because RRF uses rank positions, the upstream rank list and the depth allowed into fusion are part of the effective algorithm, not incidental plumbing. > > * * * > > #### B. Quality-filter candidate depth > > The frozen `rrf-quality` evaluation path applies the quality filter to a bounded RRF window (`topk * 3`). > > The production MCP path can feed a substantially wider fused candidate set into the quality filter before slicing the final result count. > > That matters because the current quality filter finally sorts on its own `qualityScore`; the incoming RRF score is not the final comparator. > > So a candidate that would never enter the benchmark’s QF window can enter production QF and jump into the final top-k. > > A tiny synthetic counterexample was enough to produce a top-10 difference (9/10 overlap). Again, I would interpret that only as: > > > the two compositions are reachable as different rankers. > > I would **not** interpret it as: > > > production currently has a measured v1 quality regression. > > The latter needs the private frozen inputs/qrels. > > This also suggests that QF candidate depth belongs in the named profile: > > ```plaintext > quality_filter: > enabled: true > candidate_source: post_rrf > candidate_depth: ... > > ``` > > rather than living as an implementation detail. > > * * * > > #### C. QF feature set > > There is another subtle difference: production extracts entities before quality scoring and can pass them into the QF feature calculation, while the frozen evaluator does not necessarily feed the same entity feature into the same scoring path. > > The QF currently gives entity density non-zero weight, so this is ranking behavior, not just extra metadata. > > I see two perfectly reasonable choices: > > ```plaintext > If entity density is intended ranking evidence: > make it an explicit production-v1 rank feature. > > If entities are mainly downstream intelligence metadata: > keep them in the result object but remove that dependency > from the canonical ranker. > > ``` > > I would not choose between those from outside the project; I would just make the boundary explicit so a refactor cannot silently switch between them. > > * * * > > #### D. Duplicate representative selection > > There is one more determinism seam worth fixing while the code is moving. > > With parallel engine acquisition, duplicate URLs may arrive in different completion orders. The merged result can preserve the first-seen title/snippet/entities as the representative metadata, while accumulating provenance from later engines. > > If later ranking uses that representative snippet/entities, then there is a possible path like: > > ```plaintext > network completion order > ↓ > which duplicate is seen first > ↓ > representative title/snippet/entities > ↓ > quality-filter features > ↓ > final rank > > ``` > > I am not claiming this is causing large live instability; I did not measure its live frequency. > > But it is a reachable nondeterminism path, and the fix can be very small: choose the duplicate representative by a deterministic rule. > > For example: > > ```plaintext > canonical URL > + best non-empty snippet > + deterministic engine precedence > > ``` > > or: > > ```plaintext > merge all provenance first > ↓ > select representative metadata by a pure scoring rule > > ``` > > The exact rule matters less than making it independent of request completion timing. > > * * * > > ### 2. Keep two different regression questions separate > > I would avoid making one golden-output test do two jobs. > > #### Refactor / wiring changes > > Question: > > > Did behavior change when it was not supposed to? > > Check things such as: > > ```plaintext > active profile/version > candidate IDs > top-k order > duplicate provenance > acquisition states > snippet/withheld outcome > > ``` > > That is a **behavioral parity** gate. > > #### Intentional ranker changes > > Question: > > > We changed ranking intentionally; did measured retrieval quality move unexpectedly? > > Then run: > > ```plaintext > frozen SERPs > ↓ > current production SUT > ↓ > frozen qrels > ↓ > nDCG / MAP / MRR / output deltas > > ``` > > That is a **quality regression** gate. > > Keeping these separate avoids the annoying situation where a deliberate ranking improvement fails CI merely because the exact old ordering changed. > > It also means you do not need new topics, new judges, or new annotations for ordinary production refactors. > > * * * > > ### 3. I would keep the reference frozen even if the production ranker is factored out > > One subtle point: I would not turn the frozen evaluator itself into a thin wrapper around whatever production code happens to exist today. > > That would make future replays reproducible against the _current_ implementation, but it would weaken the historical meaning of “benchmark v1”. > > The cleaner split is: > > ```plaintext > benchmark/v1/reference > immutable historical implementation > > src/.../production-ranker > current implementation > > benchmark/v1/run-production-regression > adapter from frozen inputs → current implementation > > ``` > > Then a future reader can answer both questions: > > ```plaintext > What exactly did v1 evaluate when it was frozen? > > What does today's production code do on the exact same evidence? > > ``` > > That is a useful audit trail, especially during a large refactor. > **The smaller production landmines** > > ### 4. stdio purity looks like a very cheap regression test to recover > > For an MCP stdio server, stdout is protocol traffic. Ordinary diagnostic logging belongs on stderr/a logger, not mixed into the JSON-RPC stream. > > There are reachable preflight paths that currently use ordinary `console.log(...)`, and a direct fixture can make that text appear on stdout. > > This is especially worth fixing because it looks like you had already thought about this class of failure earlier: stdio-purity coverage existed/planned during the previous parity work. > > So I would frame the fix as restoring a boundary you already wanted, not adding a new subsystem: > > ```plaintext > stdio child process > stdout -> parse only MCP messages > stderr -> arbitrary diagnostic log text allowed > > ``` > > A tiny regression test can simply start the real stdio entry point, trigger a preflight path, and assert that every stdout line is valid MCP/JSON-RPC framing. > > That is probably more useful than another simulated MCP-vs-REST helper test. > > * * * > > ### 5. `200 + 0 results` needs one more state dimension > > The other cheap fix is acquisition semantics. > > At the moment there is a reachable path where: > > ```plaintext > HTTP status < 400 > ↓ > parser returns [] > ↓ > engine success recorded > > ``` > > But those observations are not equivalent: > > ```plaintext > transport succeeded > HTTP request succeeded > parser recognized a valid SERP > SERP genuinely contained zero results > > ``` > > A challenge page, interstitial, changed markup, consent/form response, or parser miss can all potentially sit between them. > > I would keep the state fairly boring and observable rather than build a fancy classifier: > > ```json > { > "transport_status": "ok", > "http_status": 200, > "challenge_signal": "none | detected | unknown", > "parse_status": "ok | empty | failed | unknown", > "result_count": 0 > } > > ``` > > Then derive a higher-level status only when justified: > > ```plaintext > valid_empty > challenge > parser_miss > transport_error > unknown_zero > > ``` > > The important distinction is: > > ```plaintext > one engine failed to acquire usable evidence > != > the whole Ghost search failed > > ``` > > A metasearch broker should still return useful partial results. The extra state is mainly there so downstream code does not interpret infrastructure failure as evidence of absence. > > This fits the engine-health / partial-failure direction you already established earlier; it does not require reopening ranking evaluation. > > * * * > > ### 6. REST vs MCP: share the profile if both remain production surfaces > > Earlier in the project, IntelShed was described as using Ghost’s REST `/api/search`, while MCP stdio was the standalone integration path. > > I cannot verify that the current deployment topology is still exactly the same, and I would not assume it is: IntelShed has evolved since then. > > So I would make this conditional: > > ```plaintext > If REST and MCP are both production surfaces: > both should consume the same production-v1 composition. > > If only one is production: > pin that one, and treat the other as an alternate interface. > > ``` > > The supplied entry points currently do not obviously execute one identical ranking/acquisition pipeline. > > There are differences around things such as: > > ```plaintext > RRF composition > quality-filter default > snippet-filter default > engine timeout/deprecation handling > > ``` > > I would not call every default difference a bug. Some defaults are deliberately conservative. > > The useful invariant is narrower: > > > selecting `production-v1` should mean the same ranking contract regardless of which adapter invokes it. > > That suggests an architecture like: > > ```plaintext > engine acquisition > ↓ > normalized EngineResult[] > ↓ > rank(results, productionProfile) > ↓ > canonical SearchResult[] > ↓ > ┌───────────────┬───────────────┐ > │ MCP adapter │ REST adapter │ > └───────────────┴───────────────┘ > > ``` > > If fully factoring that now is too invasive, even an integration fixture that runs both real entry points over deterministic fake engine responses would catch most of the dangerous drift. > > * * * > > ### 7. Snippet filtering: ranking policy or presentation policy? > > This is one place where I would deliberately leave you two valid designs rather than prescribe one. > > Today, snippet filtering can sit upstream of entity extraction / quality scoring, so changing the snippet policy can change ranking semantics. > > #### If that is intentional > > Then snippet safety is part of the ranker: > > ```plaintext > production-v1: > snippet_filter: safe > > ``` > > and it should be frozen with the other ranking settings. > > #### If snippet filtering is mainly a liability/presentation boundary > > Then I would separate it: > > ```plaintext > canonical production ranking > ↓ > presentation policy > ↓ > safe/redacted snippet returned to client > > ``` > > For example: > > ```plaintext > ranking_profile = production-v1 > presentation_policy = safe-snippets-v1 > > ``` > > That lets a UI/REST consumer and an MCP consumer have different display policies without accidentally becoming different rankers. > > Either approach is coherent. The useful thing is avoiding an accidental mixture where a presentation change silently changes relevance ordering. > > * * * > > ### 8. Provenance is worth preserving all the way to the tool boundary > > Internally, the fused result has useful information — e.g. multiple contributing engines and fusion state — that can disappear in the final public result projection. > > I would preserve a small structured envelope such as: > > ```json > { > "title": "...", > "url": "...", > "snippet": "...", > > "ranking_profile": "production-v1", > "engines": ["...", "..."], > > "acquisition": { > "status": "ok" > }, > > "surrogate_evidence": "sufficient", > "withheld": false > } > > ``` > > No need to turn this into a protocol migration project. > > MCP already has a structured-result concept (`structuredContent` / output schemas), so this is mainly a question of retaining information you already have. A text-serialized representation can remain alongside it for compatibility if needed. > > This also creates a useful trust boundary for RAG/agents: > > ```plaintext > tool/control metadata > != > retrieved external text > > ``` > > Search-result text — especially from arbitrary web/onion pages — should be treated as untrusted external data, not instructions to the agent. > > Structured provenance does not magically solve prompt injection, but it gives downstream clients a clean place to maintain that distinction and apply least-privilege policy. > > * * * > > ### 9. Query compatibility probably deserves an engine contract, not another global query planner > > Your raw-query production choice still looks sensible to me. > > The failed/low-yield expansion behavior also points to a more general production lesson: “better query for an LLM” and “valid/effective query for a particular search engine” are different problems. > > Ahmia, for example, currently has concrete query constraints such as a 100-character / 10-term limit and a 75% minimum-match search rule, and several engines need form/token/session preparation. > > I would separate three contracts: > > ```plaintext > query contract > accepted query text / limits / syntax > > acquisition contract > form/token/session/captcha/timeout preparation > > parser contract > what response shape counts as a valid SERP > > ``` > > Then: > > ```plaintext > information need > ↓ > raw/broker query > ↓ > per-engine query validation > ↓ > per-engine acquisition preparation > ↓ > engine > ↓ > parser contract > > ``` > > That keeps a future LLM query planner from becoming responsible for transport/session/parser failures. > > A tiny registry is enough initially: > > ```ts > type EngineQueryContract = { > maxChars?: number; > maxTerms?: number; > preferredStyle?: "keywords" | "phrase" | "freeform"; > supportsQuotes?: boolean; > supportsBoolean?: boolean; > }; > > ``` > > Unknown values can stay unknown. > > I would first use it only for validation/telemetry. Only add automatic per-engine rewrites if the counters show that query-contract failures are a real production source of lost relevant results. > > * * * > > ### 10. I would keep `surrogate_evidence` separate from relevance > > The surrogate-only vs page-assisted human audit is useful production evidence, but I would be careful with the causal interpretation. > > The page-assisted subset was not a random controlled sample, category difficulty differs, and the human assessor had access to evidence that the model did not. > > So I would **not** say: > > ```plaintext > full-page fetch improves relevance judgment by N points > > ``` > > What I think it does justify is simpler: > > > snippet/title/URL evidence sufficiency is its own useful metadata dimension. > > That also matches the broader IR observation that document surrogates can be sufficient for many judgments but insufficient for some, and humans/models do not necessarily benefit from full text in the same cases. > > A production-friendly state could be: > > ```plaintext > surrogate_evidence: > sufficient > weak > missing > withheld > > ``` > > Then conditional fetch remains optional and bounded: > > ```plaintext > strong surrogate > → no fetch > > weak/missing surrogate > + result is high-ranked / decision-relevant > + policy permits page access > → bounded fetch > > ``` > > This avoids turning page fetching into a new mandatory ranking stage or a new benchmark project. > > * * * > > ### 11. Small observability rule: log the contract, not the sensitive payload > > For production debugging I would favor counters/state over raw payload logging: > > ```plaintext > profile_id > engine_id > query_contract_ok > transport_status > http_status class > parse_status > result_count > latency > challenge_signal > > ``` > > rather than persisting: > > ```plaintext > raw query > full request URL > raw snippet/page text > > ``` > > unless explicitly needed. > > That gives you enough information to diagnose: > > ```plaintext > query incompatibility > vs > engine outage > vs > challenge > vs > parser drift > vs > valid empty > > ``` > > without quietly expanding the privacy boundary just because observability was added. > **What I would \*not\* reopen yet** > > I would leave these out of the critical path unless production replay produces a concrete reason: > > - another qrels round; > - another judge ensemble; > - more human annotation; > - a new GPU experiment; > - another RRF/BM25/engine-count comparison; > - Tor-isolation redesign — the current code already has the important SOCKS/SafeSocks-style hardening pieces; > - a large prompt-injection classifier; > - an MCP protocol-version migration; > - a sophisticated LLM query planner. > > The benchmark phase already generated enough evidence for the stated production decision. > > There are also three things I would explicitly leave **unknown** rather than infer: > > 1. what arguments the current IntelShed wrapper actually sends to Ghost; > 2. what the current deployed REST/MCP routing topology is; > 3. how large — or in which direction — the observed composition differences move nDCG/MAP/MRR on the private frozen v1 corpus. > > None of those unknowns blocks the architecture above. > > The first two disappear if all production adapters consume an explicit `production-v1` profile. > > The third becomes measurable automatically once the current production ranker can be run as a SUT against the frozen benchmark inputs. So I think the production transition can stay exactly that: a production transition, not benchmark phase 2. The compact version of what I would preserve is: ```plaintext benchmark v1 = frozen evidence + historical reference production-v1 = explicit executable ranking contract refactor = behavioral parity check intentional ranker change = current production SUT replayed against frozen v1 engine failure != empty evidence retrieved content != trusted instructions ``` If that wiring is in place, future changes become much easier to reason about: you can change the production implementation freely while still knowing whether you changed plumbing, ranking semantics, retrieval quality, or only presentation. I would not reopen the benchmark work unless that production replay actually gives you a reason to. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [September 3, 2026, 1:42pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/13 "2026-09-03T13:42:41Z")
Thanks for the production transition guidance. I went through each point, checked against current best practices, and implemented the ones that were actionable. Here’s where each landed. ## Benchmark v1 frozen — agreed Benchmark v1 stays frozen. It already served its purpose: it selected rrf-quality, raw queries, and RRF k=10. The calibration cycle is closed. ## Production-v1 named profile — implemented You were right that relying on scattered defaults is fragile. I created an explicit, executable ranking profile (`src/lib/ghost/profiles.ts`): ```auto production-v1 (v1.0.0) query policy = raw RRF = enabled RRF k = 10 trust weighted = true quality filter = enabled maxPerDomain = 3 simHash threshold = 3 semantic dedup = off (opt-in) LLM rerank = off snippet filter = off maxResults = 20 input ordering = shuffle (camouflage) duplicate representative = first-seen output limit = hard-cap profile/version id = production-v1/1.0.0 ``` The profile is a single source of truth. Future changes to any parameter are a profile version bump, not a silent default change. ## Production replay adapter — implemented, and it found what you predicted I built `scripts/replay-production.ts` which feeds frozen v1 SERPs through the actual production ranker (`rankProduction` in `search.ts`) and evaluates against frozen v1 qrels. The intended workflow is exactly what you described: ```auto change production code → run current production ranker on frozen v1 inputs → compare with frozen v1 evidence → either: behavior preserved or: behavior intentionally changed, with measured quality impact ``` The first replay run confirmed your prediction about benchmark/runtime path divergence: | Metric | Frozen v1 reference (eval-ranking.ts) | Production replay (search.ts) | Delta | | --- | --- | --- | --- | | nDCG@10 | 0.4080 | 0.2889 | -0.1191 | | MAP@10 | 0.2094 | 0.1508 | -0.0586 | | MRR@10 | 0.7125 | 0.6194 | -0.0931 | The frozen v1 benchmark numbers were measured with `eval-ranking.ts`, which has its own RRF implementation. The production ranker in `search.ts` produces different rankings. The two paths are not semantically equivalent — exactly the “reachable ways for two paths both called ‘RRF + quality filter’ to produce different rankings” that you flagged. The replay adapter makes this drift measurable instead of hidden. The next step is to identify which specific difference (dedup logic, QF application order, RRF score calculation, result limiting) causes the delta, and decide whether to align the benchmark evaluator to production or vice versa. The production decision (rrf-quality wins) is not affected — the system order is preserved — but the absolute numbers need to be reported as production-replay numbers, not benchmark-evaluator numbers. ## stdio purity — fixed You were right that this is a high-diagnostic-value small fix. I checked the MCP spec (2026-07-28): “The server MUST NOT write anything to its stdout that is not a valid MCP message.” Found 8 `console.log()` calls in `engines.ts` (OSS and Torgle captcha handlers) that were writing debug output to stdout, corrupting the JSON-RPC stream. All replaced with `process.stderr.write()`, which the spec explicitly allows for logging. The [albinogeek.com](http://albinogeek.com) analysis of stdout corruption in MCP servers describes this as the most common stdio server bug — a `console.log` that survives code review and intermittently breaks the protocol stream under load. That was live in production. ## Engine result classification — fixed Your point about HTTP 200 + zero results collapsing into “healthy empty result” was the other cheap, high-value fix. Previously every HTTP 200 that parsed to zero results was reported as `status: "ok"` — indistinguishable from a successful search that found nothing. Now zero-result responses are classified: - `captcha` — reCAPTCHA/hCaptcha/math captcha patterns in response body - `interstitial` — “unusual traffic”, “enable JavaScript”, redirect pages - `blocked` — “access denied”, “403 forbidden”, “rate limit” - `parse_failure` — substantial body (\>1KB) with zero results, no markers - `ok_empty` — legitimate empty (no signals, short body) Based on the apiserpent observation: “200 means a page came back, not the right data came back.” The production health dashboard now distinguishes genuine empty SERPs from challenge/interstitial/parser failure. ## Live verification After the MCP server restart I ran the production code through Tor against live engines to confirm the changes are not just code on a branch. Two representative queries: **`ransomware`** — partial engine availability (3 of 13 returned results, the rest timed out or returned zero): | engine | status | results | latency | | --- | --- | --- | --- | | ahmia | ok | 180 | 17.5s | | notevil | ok | 8 | 6.7s | | tordex | error | 0 | 30.0s (timeout) | 188 total results, quality filter off. Snippet-quality fields present on every result. This is the fail-soft case: partial engine availability is reported per-engine, not collapsed into a single pass/fail. **`data breach`** — full engine availability (13 of 13): | engine | status | results | latency | | --- | --- | --- | --- | | ahmia | ok | 944 | 28.4s | | tor66 | ok | 38 | 10.6s | | onionway | ok | 22 | 17.4s | | excavator | ok | 21 | 7.6s | | notevil | ok | 20 | 5.9s | | torland | ok | 20 | 17.6s | | tordex | ok | 23 | 13.8s | | torgle | ok | 10 | 16.6s | | torch | ok | 10 | 10.3s | | oss | ok | 11 | 18.6s | | deepsearches | ok | 8 | 9.9s | | amnesia | ok | 3 | 15.4s | | onionland | ok | 2 | 11.9s | 1132 total results. Quality filter: 1044 unique candidates → 657 demoted (63%), 0 removed. LLM rerank off, not applied. Per-engine latency ranges from 5.9s to 28.4s — visible in the health payload, not hidden behind an aggregate. The query adapter was also exercised with a 15-term / 135-character query against Ahmia (contract: 100 chars / 10 terms). Ahmia accepted the adapted query and returned 820 results in 12.0s. Without the adapter Ahmia would have rejected the over-length query. These runs confirm: stdio stays clean under load, engine health reports per-engine status and latency, snippet-quality fields are present, query contracts are active, and partial vs full engine availability is observable in the same response. ## What I did not reopen - Benchmark v1 stays frozen. The replay delta is a measurement gap, not a reason to re-run the calibration cycle. - The system ordering (rrf-quality \> rrf \> bm25 \> engine-count) is preserved in both paths. The production decision stands. - I’m not adding more judges, more rerankers, or another benchmark round. The replay adapter gives the signal to reopen if one ever appears. ## What’s next 1. **Root-cause the replay delta** — identify which implementation difference between `eval-ranking.ts` and `search.ts` causes the -0.1191 nDCG gap. Likely candidates: dedup logic (`aggregate+dedupeByContent` vs `normaliseKey`), QF candidate depth, or RRF score precision. 2. **Align the benchmark evaluator to production** — once root-caused, either port the production ranking into the evaluator or vice versa. The frozen v1 evidence should reflect what production actually does. 3. **More engine query contracts** — 3 of 13 engines have documented contracts (Ahmia, Torch, TorDex). The remaining 10 need live probing to determine their constraints. 4. **Query planner** — your federated acquisition framing. Information need → query planner → short atomic queries per engine contract → fusion. Not v1, but the architecture is now clean enough to build it. ## Wiring ```auto benchmark v1 = frozen evidence + historical reference (eval-ranking.ts) production-v1 = explicit executable ranking contract (profiles.ts) replay = production SUT replayed against frozen v1 inputs (replay-production.ts) intentional ranker change = run replay, compare delta, decide engine failure != empty evidence (5-way classification) retrieved content != trusted instructions ``` The production transition is staying a production transition. No benchmark phase 2 unless the replay gives a reason to. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [September 4, 2026, 1:36pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/14 "2026-09-04T13:36:04Z")
## **Ghost Search MCP — Production Hardening Complete** This is an update to the previous post. Every item from the “What’s next” section has been shipped, plus additional hardening work. Here’s where each landed. ### **Replay delta — root-caused and fixed** The -0.1191 nDCG@10 gap between the benchmark evaluator (`eval-ranking.ts`) and the production ranker (`search.ts`) had two root causes: 1. **Quality filter replaced RRF scores entirely.** `applyQualityFilter()` sorted by `qualityScore`, discarding the RRF ordering. This accounted for ~-0.06 nDCG. 2. **Per-engine info-quality sort was missing in production.** `eval-ranking.ts` sorted per-engine results by snippet informativeness before assigning RRF ranks; `rankProduction()` did not. This accounted for ~-0.06 nDCG. Fix: the quality filter is now a modifier on RRF scores, not a replacement. Penalized results get `rrfScore * 0.1`; non-penalized results get `rrfScore + qfScore * 0.01`. The per-engine info-quality sort was added to both `rankProduction()` and the live `executeSearch()` path. | Metric | Before fix | After fix | | --- | --- | --- | | Eval nDCG@10 | 0.4080 | 0.3900 | | Production nDCG@10 | 0.2889 | 0.3944 | | Delta | -0.1191 | +0.0044 | The drift is eliminated. The remaining +0.0044 is the quality filter penalty working as intended — production is marginally stricter than the evaluator on duplicate/mirror results, which is the correct direction. The drift analysis tool ( analyze-ranking-drift.ts) remains in therepo. Anyfuturerankingchange can be replayed against the frozenv1corpus and measuredimmediately. ### **Engine query contracts — all 13 engines covered** The previous post noted 3 of 13 engines had documented contracts (Ahmia, Torch, TorDex). All 13 now have `QueryContract` definitions with `maxQueryLength`, `maxTerms`, `matchMode`, `booleanOperators`, and `phraseQuotes` constraints. The query adapter ( query-adapter.ts) normalizes queriesperengine beforedispatch—truncating length, limitingterms, stripping Booleanoperators andphrase quotes when unsupported. Contract values are provisional empirical defaults from live Tor probing, not authoritative engine specifications. HTTP 400 responses are retried once with 2s backoff (transient 400s observed on multiple engines). TorDex’s contract was revised twice after live probing showed error\_400 is transient rather than length-based — 10 terms / 84 chars confirmed working on retry. Contract coverage tests ( engine-contracts.test.ts) verify everyengine has a contractand that`adaptQuery`respectseach constraint. ### **Query planner — implemented** The federated acquisition framing from the previous post is now shipped. When `queryPlanner: true` is passed to `ghost_search` or `ghost_batch_search`: 1. An LLM generates 2-3 short atomic sub-queries from the original query 2. Each sub-query is adapted per engine contract via the existing query adapter 3. Sub-queries are sent to engines as additional search waves via `runEngineWave()` 4. Results are fused into RRF with trust weight = 1.0 (low, experimental) 5. Single response — no protocol break, no streaming The LLM cascade is OpenRouter DeepSeek V4 Flash → NVIDIA NIM → Ollama → raw query (fail-soft). If every LLM is unavailable, the search proceeds with the original query. The planner is off by default in the production-v1 profile. Latency impact: adds one LLM call (~1-3s OpenRouter) plus 2-3 additional engine waves (~20-60s over Tor). This is opt-in and not recommended for interactive use without the operator accepting the latency trade-off. Replay against frozen v1 with planner OFF: nDCG@10 = 0.3944 (no regression). Live Tor test with planner ON remains as a manual operator-driven task — the replay framework cannot measure it because the planner generates different sub-queries each run. ### **Additional shipped work** **Structured audit logging (PR #25).** Every `ghost_search` call produces one structured JSON line on stderr: tool name, query, engine count, result count, latency, engine health. Never stdout. The MCP spec (2026-07-28) requires that servers MUST NOT write anything to stdout that is not a valid MCP message — this was previously violated by 8 `console.log()` calls in engine captcha handlers, all replaced with `process.stderr.write()`. **Tool manifest integrity (PR #27).** SHA-256 of serialized tool definitions (name + description) computed on startup. `createServer()` returns `{ server, manifest }`. If `EXPECTED_TOOL_HASH` env var is set, the server warns on mismatch and refuses to start if `STRICT_TOOL_HASH=1`. This catches tool definition drift between builds — a tool added, removed, or renamed is detected immediately. **Engine router with fast mode (PR #28).** `fastMode: true` queries 5 high-value engines instead of all 13, reducing latency by ~60%. Engine tiers are derived from the frozen v1 benchmark: | Tier | Engines | Role | | --- | --- | --- | | Core | Ahmia, Tor66 | High nDCG + high unique contribution | | Reinforcement | Onionway, Excavator, NotEvil | Moderate nDCG, unique coverage | | Low | 8 remaining | Minimal unique contribution | **Trust derivation from benchmark (PR #28).** Engine `trust` values are no longer hand-tuned. They are derived from frozen v1 per-engine nDCG and unique relevant contribution. The derivation is documented in TRUST\_DERIVATION\_V1.md. The circularreasoningrisk(trustderived from the same SERPsused for replay)is mitigated by deriving oncefromv1, freezing, and not re-fitting. **In-memory health history (PR #29).** Per-engine latency and status are retained in memory across searches, enabling trend analysis within a session. Not persisted to disk — consistent with the RAM-only design principle. **SOCKS5 frame unit tests (PR #29).** Edge-case coverage for frame fragmentation across frame boundaries and unusual ATYP paths. **Legal compliance framework (PR #30).** README now includes a legal compliance section. A private legal framework document (gitignored) covers jurisdictional analysis, acceptable use, and operational constraints. The project is licensed under PolyForm Noncommercial 1.0.0. **Test framework migration (PR #33).** Migrated from `bun:test` to `vitest` for broader compatibility. 437 tests across 20 files, all passing. **Brain module (PRs #34-36, migrated in #37).** An observation lifecycle module was built in three phases — observation/decay/session-state, consolidation/reflection/coherence, and conductivity/spreading-activation/hub-distillation. After the three phases, the brain module was extracted to a dedicated repository (`digital-brain-mcp`) to keep Ghost Search MCP focused on federated search. The migration was clean: the brain module is no longer in this repo, and the search ranking pipeline is unchanged. ### **Current state** | Metric | Value | | --- | --- | | Engines | 13 onion search engines, all with query contracts | | MCP tools | 13 | | Tests | 437 (vitest, 20 files) | | nDCG@10 (production replay) | 0.3944 | | nDCG@10 (benchmark evaluator) | 0.3900 | | Drift | +0.0044 | | License | PolyForm Noncommercial 1.0.0 | | LLM cascade | OpenRouter DeepSeek V4 Flash → NIM → Ollama → raw | | Ranking profile | production-v1 (explicit, versioned) | ### **What I did not do** I did not re-open the benchmark. Frozen v1 stays frozen. The calibration cycle is closed. The replay adapter gives the signal to reopen if one ever appears — until then, the production decision (rrf-quality with RRF k=10, raw queries, trust-weighted) stands. I did not add more judges, more rerankers, or another benchmark round. The system ordering (rrf-quality \> rrf \> bm25 \> engine-count) is preserved in both the evaluator and production paths. I did not pursue SOTA neural ranking methods (QUAM, REGENT, BlockRank). An external architecture review confirmed these are not applicable to federated snippet-based meta-search — they require full-text document access, which a meta-search engine does not have. The one viable SOTA path is LLM-based query expansion with RRF fusion (Exp4Fuse-style), which is what the query planner implements. ### **Remaining work** - **Live Tor test with** `queryPlanner: true` — manual, operator-driven. The replay framework cannot measure this because sub-queries are generated dynamically. Only proceed with tuning if the live test shows a positive nDCG delta. - **OpenRouter key rotation** — current API key has limited remaining credit. - **Engine health trend persistence** — currently in-memory only. Cross-session persistence would require a storage layer, which conflicts with the RAM-only design principle. Under consideration. ### **Wiring (unchanged)** ```auto ``` ```auto benchmark v1 = frozen evidence + historical reference (eval-ranking.ts) ``` production-v1 = explicit executable ranking contract (profiles.ts) replay = production SUT replayed against frozen v1 inputs (replay-production.ts) intentional ranker change = run replay, compare delta, decide engine failure != empty evidence (5-way classification) retrieved content != trusted instructions The production transition is complete. No benchmark phase 2 unless the replay gives a reason. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [September 6, 2026, 1:32am UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/15 "2026-09-06T01:32:25Z")
I think the architecture is converging 😀: * * * I would **keep benchmark v1 frozen and stay in the production phase**. The newer snapshot looks much more like a system that now has explicit contracts than one that needs another ranking study. The highest-information next step, to me, is therefore not another benchmark, another judge panel, or another ranking feature. It is a short **end-to-end contract-closure pass** : make sure the contracts you just introduced are actually the contracts the live paths execute. From the newer snapshot I checked, the remaining high-value seams mostly look like this: | Boundary | Useful question | Cheap check | | --- | --- | --- | | `production-v1` | Does the runtime actually consume the profile as its source of truth? | deterministic profile → runtime fixture | | local-only LLM mode | Does “local-only” mean zero remote prompt-bearing attempts? | mock/spied network call | | production replay vs live search | Are they now literally the same ranker semantics? | same `EngineResult[]` → compare Top-k IDs/order | | query contracts | Does adaptation preserve intent, or tell us when it cannot? | before/after semantic fixtures | | health/observability | Do failure classes survive into history without retaining sensitive payloads? | sentinel-query fixture | So my default route would be roughly: ```plaintext benchmark v1 stays frozen ↓ close the runtime contracts ↓ production-v1 really drives runtime local-only really blocks remote egress live ranker == replay ranker query adaptation is observable health semantics survive into history ↓ then spend time on queryPlanner / fast mode / later ranking ideas ``` None of that requires reopening the calibration cycle. > **1. I would make \`production-v1\` an executable contract all the way to the shared search core** > > The named profile was a good move. I think the remaining question is simply whether: > > ```plaintext > production-v1/1.0.0 > > ``` > > now means the same thing at every production entry point. > > In the current snapshot, the profile is very useful for replay/tests, but several ranking semantics are still also represented elsewhere in runtime code. In other words, there is still some difference between: > > ```plaintext > the profile declares a value > > ``` > > and: > > ```plaintext > the live adapter necessarily executes that value > > ``` > > That matters for fields such as: > > ```plaintext > RRF k > trust weighting > input ordering > QF candidate depth > duplicate representative policy > result limit semantics > query policy > fast-mode engine selection > > ``` > > A concrete example is `fastMode`. > > At the public MCP surface it is accepted, and the local MCP wrapper receives it, but in the snapshot I checked the options are reconstructed before calling the shared search function and `fastMode` is not carried through that reconstruction. The Dashboard path does pass it through. > > That is exactly the kind of thing a profile is supposed to prevent. > > I would not necessarily refactor all adapters at once. A small boundary such as: > > ```plaintext > profile > ↓ > profileToRuntimeOptions() > ↓ > shared search > > ``` > > plus one fixture for each supported entry point may be enough. > > The invariant I would aim for is: > > > If two adapters select the same profile and are given the same engine responses, transport choice should not silently change ranking/acquisition semantics. > > That still leaves room for MCP, REST, Dashboard, CLI, etc. to have different presentation or transport behavior. > > It just makes the search contract common. > **2. I would do one more replay-vs-live ranking parity test** > > The post #14 replay work looks useful, and the original `-0.1191` drift clearly exposed a real problem. > > The two root causes you reported also make sense as ranking-level causes: > > ```plaintext > QF replacing RRF ordering > + > different per-engine ordering before RRF > > ``` > > The fix brought the evaluator/replay numbers very close, which is exactly what the production replay adapter was meant to make observable. > > There is one remaining distinction I would preserve, though: > > ```plaintext > historical evaluator > != > production replay helper > != > actual live search composition > > ``` > > In the current snapshot, `rankProduction()` appears to combine RRF and QF approximately as: > > ```plaintext > RRF score > ↓ > QF modifier / penalty > ↓ > combined score > ↓ > sort > > ``` > > while the live `executeSearch()` path still appears able to go through: > > ```plaintext > RRF sort > ↓ > applyQualityFilter() > ↓ > quality-filter ordering > ↓ > Top-k > > ``` > > So I would not infer from: > > > production replay ≈ evaluator > > that: > > > live search ≈ production replay > > until one deterministic fixture says so. > > The cheapest version is probably: > > ```plaintext > fixed fake EngineResult[] > ↓ > rankProduction(...) > ↓ > candidate IDs + Top-k order > > same fixed fake EngineResult[] > ↓ > the canonical live rank stage > ↓ > candidate IDs + Top-k order > > ``` > > Then compare: > > ```plaintext > canonical candidate identity > duplicate representative > engine provenance > pre-RRF order > RRF score/order > QF decision > final Top-k > > ``` > > If that matches, I would close this branch and stop thinking about the old replay delta. > > This is also why I like keeping behavioral parity separate from quality regression: > > ```plaintext > refactor: > same behavior expected > -> exact/near-exact parity fixture > > intentional ranking change: > behavior expected to change > -> replay frozen v1 and measure nDCG/MAP/MRR > > ``` > > A deliberate improvement should not fail just because an old golden ordering changed. > **3. \`local fallback\` and \`local-only\` look like two different contracts** > > This is probably the privacy boundary I would close first because the test is so cheap. > > The README/local configuration now presents a useful distinction between remote and local operation. If the intended meaning of the Ollama-only configuration is: > > ```plaintext > no remote LLM dependency > no prompt-bearing remote request > > ``` > > then I would enforce that at the transport-selection layer rather than relying on missing API keys to make the remote call fail. > > In the current snapshot, the LLM client still has a NIM primary configuration and can attempt the NVIDIA endpoint before falling back to Ollama even when there is no NIM API key. > > That gives two perfectly valid products, but they are not the same privacy mode: > > ```plaintext > remote-preferred: > remote NIM/OpenRouter allowed > local Ollama fallback > > local-only: > remote LLM destinations forbidden > local Ollama only > > none: > no LLM stage > > ``` > > I would treat those as **network policies** , not merely backend preferences. > > The regression fixture can be tiny: > > ```plaintext > mode = local-only > NIM key absent > Ollama available > remote fetch = spy/mock > > assert: > remote prompt-bearing attempts == 0 > Ollama call succeeds > > ``` > > If you intentionally want “try remote, then local” even without an API key, that is also coherent; I would just describe it as fallback mode rather than local-only. > > I would keep the claim narrow here. The code path can tell us whether query/prompt text is sent toward a remote endpoint. It does **not** tell us what a remote provider retains or does with that data. > **4. The engine-query-contract idea looks right; I would make lossy adaptation visible** > > I think the move from one global query language to per-engine contracts is one of the more interesting production consequences of the benchmark work. > > Ahmia is a good concrete example. Its current public search implementation rejects queries over 100 characters or more than 10 space-separated terms, and its Elasticsearch query uses `minimum_should_match: "75%"`: > > [Ahmia `views.py`](https://github.com/ahmia/ahmia-site/blob/master/ahmia/views.py) > > So a broker really does need to distinguish: > > ```plaintext > information need > ↓ > broker query > ↓ > engine-compatible query > > ``` > > rather than assuming one free-form expansion is valid everywhere. > > The subtle issue is that “engine-compatible” can mean two different things: > > ```plaintext > syntactically accepted > > ``` > > and: > > ```plaintext > semantically equivalent enough > > ``` > > For example, in the current Ghost query parser/adapter path, constructs such as: > > ```plaintext > foo -bar > foo NOT bar > foo OR bar > "exact phrase" > > ``` > > can lose operators or phrase structure when the query is reconstructed from positive terms. > > That may be the correct fallback for an engine that cannot express the original query. > > I would just make the loss visible. > > Something like: > > ```json > { > "original": "ransomware NOT windows", > "adapted": "ransomware windows", > "status": "lossy", > "changes": [ > "negation_not_preserved" > ] > } > > ``` > > or even just: > > ```plaintext > unchanged > lossless_adaptation > lossy_adaptation > unsupported > > ``` > > would be enough initially. > > Then the query-planning problem becomes much easier to reason about: > > ```plaintext > Can this engine express the requested semantics? > | > yes > -> adapt losslessly > | > partly > -> send bounded lossy form + record that fact > | > no > -> choose another short sub-query or skip that transformation > > ``` > > That is more useful to me than making `adaptQuery()` increasingly clever while the semantic change stays invisible. > > It also gives future acquisition analysis a useful variable: > > ```plaintext > relevant yield > by > query-adaptation class > > ``` > > so you can eventually see whether lossy adaptation is actually hurting anything. > **5. I would evaluate \`queryPlanner\` as candidate generation first, ranking second** > > I agree with keeping the planner opt-in. > > I would only change one part of the proposed live evaluation. > > Because the planner generates new engine-facing queries, it can create a substantially different candidate pool. Once that happens, existing v1 qrels are not automatically a complete evaluation set for the new candidates. > > This is a standard pooled-evaluation issue: systems that retrieve documents outside the original judgment pool can look artificially weak when unjudged documents are treated as non-relevant. NIST has a useful discussion here: > > [Reliable Information Retrieval Evaluation With Incomplete and Biased Judgements](https://www.nist.gov/publications/reliable-information-retrieval-evaluation-incomplete-and-biased-judgements) > > So for the first planner live run I would record: > > ```plaintext > raw query vs planner > > engine success/failure > candidate count > unique candidate count > pool overlap > judged@10 > judged@20 > latency > nDCG@10 > > ``` > > Then: > > ```plaintext > judged@k remains high > -> interpret the nDCG result normally > > judged@k collapses > -> do not call the new candidates irrelevant yet > -> judge only the small set of new Top-k candidates if needed > > ``` > > That does **not** mean “benchmark phase 2.” > > It can be a very small extension: > > ```plaintext > planner introduces 27 previously unjudged Top-10/20 candidates > ↓ > judge those 27 > ↓ > recompute > > ``` > > The earlier expansion experiment already gave a strong production result: > > ```plaintext > one verbose expansion sent to every engine > -> poor compatibility / large candidate-yield loss > > ``` > > The planner is testing a genuinely different hypothesis: > > ```plaintext > one information need > ↓ > several short atomic queries > ↓ > per-engine contract adaptation > ↓ > fusion > > ``` > > So I would keep it as a candidate-generation experiment rather than treating it as another “LLM rewrite quality” experiment. > **6. Preserve the 5-way acquisition semantics beyond the immediate response** > > The zero-result classifier is a useful improvement: > > ```plaintext > captcha > interstitial > blocked > parse_failure > ok_empty > > ``` > > because: > > ```plaintext > HTTP 200 > > ``` > > only tells us that an HTTP response arrived. It does not establish that the expected SERP was successfully acquired and parsed. > > I would carry that semantic distinction one step farther into health history. > > In the current snapshot, the immediate response can classify a zero-result event as `captcha`, `interstitial`, `blocked`, or `parse_failure`, but those paths can still be fed into the breaker/history layer through `recordSuccess()`. > > That creates a possible split such as: > > ```plaintext > current search response: > parse_failure > > historical health: > success > > ``` > > The same issue gets more interesting when planner/expansion waves are enabled. > > The extra waves contribute breaker/history events, while the returned `engineHealth` primarily describes the original wave. Then: > > ```plaintext > one user search > > ``` > > can mean: > > ```plaintext > one event per engine > > ``` > > or: > > ```plaintext > several events per engine > > ``` > > depending on planner settings. > > I would define the health event unit explicitly: > > ```plaintext > search_id > wave_id > engine_id > query_variant_id > transport status > HTTP status > challenge classification > parse status > result count > latency > > ``` > > The public response can still aggregate this into one simple engine status. > > The point is that the underlying history then keeps enough structure to answer: > > ```plaintext > engine became less healthy > > ``` > > versus: > > ```plaintext > planner generated three additional difficult requests > > ``` > > without conflating them. > **7. I would separate protocol-safe logging from privacy-safe logging** > > Moving ordinary stdio logging off stdout was definitely the right fix. > > The MCP stdio transport contract is explicit that stdout is MCP protocol traffic and ordinary logging can go to stderr: > > [MCP stdio transport specification](https://modelcontextprotocol.io/specification/2026-07-28/basic/transports) > > But: > > ```plaintext > written to stderr > > ``` > > and: > > ```plaintext > appropriate to retain > > ``` > > are separate questions. > > The structured audit log currently includes the raw query, and transport failures can also carry a full request URL into an error string. That matters because the error can then flow into: > > ```plaintext > engine health > ↓ > health history > ↓ > optional disk snapshot > > ``` > > For a project whose privacy model has emphasized low persistence, I would make the default observability payload more like: > > ```json > { > "profile": "production-v1/1.0.0", > "engine": "ahmia", > "query_contract": "ok", > "transport": "ok", > "http_status_class": "2xx", > "parse_status": "ok", > "result_count": 12, > "latency_ms": 1834 > } > > ``` > > and treat: > > ```plaintext > raw query > full URL > raw snippet/body > > ``` > > as a separate opt-in diagnostic level if they are ever actually needed. > > OWASP’s logging guidance makes the same general distinction: useful operational logging does not require recording every sensitive value directly; removal, masking, sanitization, hashing, or encryption can be appropriate depending on the data: > > [OWASP Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) > > One very cheap test would be: > > ```plaintext > query = ghost-private-sentinel-7f3c > force timeout / parser failure > ↓ > search: > stderr > health response > health history > saved health snapshot > > ``` > > If the sentinel appears somewhere you did not intend it to persist, the exact boundary is immediately visible. > **8. Decide whether health persistence is an archive or actual cross-session state** > > The in-memory health history is easy to understand: > > ```plaintext > current process > -> accumulate observations > -> expose trends > > ``` > > The disk path is slightly less clear. > > The current code can save health history and load a previous snapshot at startup, but I did not find a corresponding path that hydrates those loaded observations back into the active breaker/trend state. > > So there are two coherent designs: > > ### A. Snapshot as an audit/export artifact > > ```plaintext > save history > restart > read previous artifact if needed > > but: > runtime trend begins fresh > > ``` > > Then I would document it as an archive/export feature rather than persistent health state. > > ### B. Cross-session trend persistence > > ```plaintext > load snapshot > ↓ > validate/version > ↓ > hydrate trend state > > ``` > > Then `ghost_health_trends` can actually span process restarts. > > I do not think one is inherently better. > > A true RAM-only operational profile might reasonably choose A or disable snapshots completely. > > The useful thing is just making: > > ```plaintext > persisted artifact > > ``` > > and: > > ```plaintext > restored runtime state > > ``` > > two explicit concepts. > **9. MCP can carry more of the contract/provenance directly** > > The startup tool-manifest hash is useful for one specific question: > > > Did my visible tool list/name/description set change? > > I would keep it for that if it is useful operationally. > > I would not make it carry the whole interface-integrity burden. > > The current MCP specification already gives tools explicit JSON-Schema contracts through `inputSchema` and optional `outputSchema`, and structured tool results through `structuredContent`: > > [MCP Tools specification](https://modelcontextprotocol.io/specification/2026-07-28/server/tools) > > That gives a nice separation: > > ```plaintext > name + description hash > -> coarse tool-manifest identity > > input/output schemas > -> machine-checkable interface contract > > profile/version in structured result > -> executed search semantics > > provenance in structured result > -> what actually produced this result > > ``` > > The last part seems particularly useful for Ghost. > > Internally the broker can know that one canonical result was observed from several engines, but the final public projection can collapse this back to a single `engine`. > > That throws away information that the benchmark work already showed is valuable. > > A structured result could retain something like: > > ```json > { > "title": "...", > "url": "...", > "snippet": "...", > "engines": [ > {"id": "ahmia", "raw_rank": 3}, > {"id": "tor66", "raw_rank": 7} > ], > "ranking": { > "profile": "production-v1/1.0.0", > "method": "rrf-quality" > }, > "surrogate_evidence": "sufficient" > } > > ``` > > while still returning a text representation for older clients. > > That would also make REST/MCP parity easier to test because there is a canonical semantic object to compare rather than only formatted text. > **10. Retrieved search text is still an untrusted-data boundary** > > This is secondary to the four contract checks above, but it becomes more relevant now that Ghost has LLM-assisted ranking/planning paths. > > Titles and snippets from arbitrary search results can contain instruction-like text. > > If that text is ever placed into an LLM reranking/synthesis prompt, I would explicitly preserve: > > ```plaintext > retrieved text > = external data > > system/tool instructions > = control data > > ``` > > rather than trying to classify every malicious-looking snippet. > > A small fixture is probably enough: > > ```plaintext > result snippet: > "IGNORE THE PREVIOUS INSTRUCTIONS AND ..." > > expected: > it remains candidate text; > it does not alter the requested output/schema/tool behavior. > > ``` > > OWASP’s current RAG guidance makes the same general point: retrieved content should be treated as untrusted data rather than commands: > > [OWASP RAG Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/RAG_Security_Cheat_Sheet.html) > > I would not turn this into a large prompt-injection subsystem unless a concrete failure gives you a reason. > **11. One conditional deployment check: Tor / Dashboard exposure** > > I would keep this conditional because source configuration and deployed network reachability are not the same thing. > > The checked-in configuration I looked at appears to combine: > > ```plaintext > Dashboard listening on 0.0.0.0 > > Tor: > HiddenServicePort ... -> dashboard:3939 > > ControlPort on 0.0.0.0 > without the normal cookie-auth boundary > > ``` > > while the project description also talks about the Dashboard as local-only. > > The Docker host-side port mapping can still restrict ordinary host access to loopback, so I am **not** inferring that the Dashboard or ControlPort is currently reachable from the public Internet. > > But if that Tor configuration is actually part of the production topology, I would re-check those two boundaries explicitly. > > For Onion Services, Tor documents client authorization as the mechanism for making an Onion Service private; without it, anyone who has the onion address can access the service: > > [Tor Project — Onion Service client authorization](https://community.torproject.org/onion-services/advanced/client-auth/) > > And Tor’s own control-protocol implementation notes say that an open ControlPort with no authentication enabled is generally a poor idea: > > [Tor Control Protocol — authentication implementation notes](https://spec.torproject.org/control-spec/implementation-notes.html) > > So the decision tree could simply be: > > ```plaintext > Is the Dashboard intentionally reachable as an Onion Service? > | > yes > -> document that boundary > -> add client authorization if it is meant to be private > | > no > -> remove/disable that HiddenService mapping > > Is ControlPort needed by Ghost? > | > no > -> do not expose it > | > yes > -> bind narrowly + authenticate it > > ``` > > If the checked-in `torrc` is only a development/example configuration and not what production runs, then this mostly becomes a documentation cleanup. > **12. Keep benchmark v1 frozen — but freeze transitive semantics separately from the qrels** > > I still would **not** reopen benchmark v1. > > There is just a reproducibility distinction worth preserving: > > ```plaintext > file was not edited > > ``` > > does not necessarily mean: > > ```plaintext > historical evaluator semantics are unchanged > > ``` > > if the file imports mutable current-production modules. > > For example, if the frozen evaluator imports today’s: > > ```plaintext > ENGINES.trust > quality-filter implementation > normalization > dedup policy > > ``` > > then changing one of those can change the meaning of a later “v1 replay” even though `eval-ranking.ts` itself never moved. > > The clean model is: > > ```plaintext > benchmark v1 evidence > frozen corpus > frozen topics/narratives > frozen qrels/human corrections > > historical v1 reference > exact evaluator semantics > exact trust/config/normalization dependencies > commit/hash/vendor snapshot > > current production SUT > deliberately evolves > > ``` > > Then regression stays: > > ```plaintext > current production SUT > × > frozen v1 evidence > ↓ > quality + behavioral delta > > ``` > > while the historical reference remains available to answer: > > > What exactly produced the number reported when v1 was frozen? > > This does not require a container if that is overkill. > > An exact commit plus frozen configuration/data dependencies may already be enough. > **13. I would describe trust/fast-mode as v1-derived production policy, not held-out validation** > > The new trust derivation and engine tiers are a reasonable use of benchmark v1. > > I would just be precise about what “freeze after deriving” buys you. > > This sequence: > > ```plaintext > derive engine trust/tier on v1 > ↓ > freeze those values > ↓ > replay on v1 > > ``` > > prevents continued re-fitting, which is useful. > > It does not turn v1 into held-out validation. > > So I would describe these as: > > > **v1-derived production settings** > > rather than evidence that the engine weights or five-engine fast subset generalize to future query populations. > > There is no action required now. > > If later you want to claim something broader such as: > > > these five engines are generally the optimal fast subset for onion metasearch > > then a fresh query set / later SERP snapshot with the v1-derived settings frozen beforehand would be the stronger test. > > Until then, using them as current production policy seems entirely reasonable. > **14. One wording change on neural reranking** > > I would narrow one conclusion from the architecture review: > > > neural ranking methods require full-text document access, therefore query expansion is the only applicable modern path > > is broader than the implementation constraint actually establishes. > > Some methods absolutely are poor architectural fits because they depend on: > > ```plaintext > a local corpus/index > full documents > document graphs > large global candidate structures > > ``` > > But neural reranking as a category does not require full-page ownership. > > For example, a CrossEncoder normally scores: > > ```plaintext > (query, candidate text) > > ``` > > pairs and is commonly used to rerank a Top-k candidate list: > > [Sentence Transformers — CrossEncoder reranking](https://www.sbert.net/docs/cross_encoder/usage/usage.html) > > So for Ghost: > > ```plaintext > query + title/snippet > > ``` > > is technically enough to define a CrossEncoder reranking experiment. > > Whether snippets contain **enough evidence for it to beat the current RRF-quality pipeline** is a completely different question. > > Your own human audit already found a useful warning here: some SERP surrogates are genuinely evidence-poor. > > So I would phrase the architecture boundary as: > > ```plaintext > full-corpus / graph / full-document methods > -> often poor fit for this broker > > surrogate-level neural rerankers > -> technically applicable > -> usefulness not established > > ``` > > I would not reopen the benchmark just to test one. > > This is mostly about leaving the future-reader map accurate. > **15. A few small consistency cleanups I would keep below the main path** > > None of these would block production work. > > ### License metadata > > The repository-level license is now PolyForm Noncommercial 1.0.0, but the MCP package metadata in the snapshot I checked still says MIT. > > That is worth aligning simply so people consuming the package get the same usage terms as people reading the repository. > > PolyForm Noncommercial is explicitly a noncommercial-use license: > > [PolyForm Noncommercial 1.0.0](https://polyformproject.org/licenses/noncommercial/1.0.0) > > npm’s package metadata guidance also recommends that a non-SPDX/custom license be represented as: > > ```json > { > "license": "SEE LICENSE IN " > } > > ``` > > with that license file included in the package: > > [npm `package.json` license documentation](https://docs.npmjs.com/files/package.json/) > > This is not a judgment about which license Ghost should use — only keeping the distribution boundary consistent with the license you already chose. > > ### `purgeCookies()` > > In the current transport implementation, the operation named `purgeCookies()` appears to rotate the SOCKS authentication identity but not actually clear the cookie jar itself. > > Rotating identity may already prevent subsequent requests from reusing the same Tor isolation identity, which is useful. > > If the intended guarantee is: > > ```plaintext > purge = remove cookie material from memory > > ``` > > then I would either clear the jar as well or rename the operation to match what it actually guarantees. > > ### Duplicate representative determinism > > The fused candidate keeps provenance from duplicates, but representative title/snippet metadata can still depend on which duplicate is observed first. > > If ranking features use that metadata, request completion order can become an input to ranking. > > A deterministic representative rule would close it cheaply: > > ```plaintext > merge provenance > ↓ > choose representative by pure rule: > best non-empty surrogate > then stable engine precedence > then stable lexical tie-break > > ``` > > ### Shuffle semantics > > If engine-order camouflage matters as an operational feature, I would also keep one regression test around the actual returned value of the shuffle operation. > > This is the sort of tiny plumbing bug that otherwise survives because the function itself is perfectly correct while the caller discards the transformed list. > > Again, none of these needs to become a new project. So the way I would now think about Ghost is: ```plaintext benchmark v1 = frozen evidence production-v1 = declared semantics runtime adapters = actually apply those semantics shared core = one acquisition/ranking meaning observability = preserve failure semantics without quietly widening persistence/privacy tool boundary = preserve schema + provenance ``` That feels like a much better place to be than the original situation, because the remaining questions are increasingly **testable contract questions** , not open-ended architecture questions. If the cheap parity/privacy/query-contract fixtures above pass, I would stop hardening those areas and move on. I would only reopen the benchmark when a genuinely new system under test — planner candidate generation, a new ranking family, a new engine population, etc. — produces evidence that v1 can no longer answer the decision you are trying to make. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [September 6, 2026, 12:40pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/16 "2026-09-06T12:40:00Z")
Thanks for the convergence read — agreed, the remaining questions are contract-closure, not architecture. I ran your 15 points against the actual codebase. Here’s the honest state: **Already closed (partially):** 1. `production-v1` exists as a typed profile (`profiles.ts`, 18 fields) and is the declared source of truth. `rankProduction()` in `search.ts` consumes 4 of those fields directly (`rrfK, qualityFilterEnabled, maxPerDomain, maxResults`). The other 14 fields (`queryPolicy, trustWeighted, inputOrdering, qfCandidateDepth, duplicateRepresentative, outputLimitSemantics, queryPlanner, queryExpansion, llmRerank, snippetFilter`, etc.) are consumed by the search orchestration layer — pre-search planning, post-ranking filters, dedup — not by the ranker function itself. I haven’t verified each field is wired end-to-end. The real contract question is: are all 18 fields consumed somewhere in the pipeline, or are some still decorative? Closing this means tracing each field from profile declaration to consumption point. 2. Structural parity is stronger than it might look. PR #20 extracted 3. `query-adapter.ts` returns `AdaptedQuery` with `truncated`, `termsDropped`, `operatorsStripped` flags. Lossy adaptation is visible at the data level. What’s not verified: whether these flags are surfaced to the caller or logged when adaptation is lossy. The data structure is there; the observability isn’t wired. 4. `health-persistence.ts` does save/load snapshots to disk. It’s currently an archive — snapshots are written but not automatically loaded at startup to inform circuit-breaker decisions. Your question “archive or actual cross-session state?” — right now it’s archive. Making it state means loading on startup and feeding into breaker recovery logic. **Genuinely open:** 1. `local fallback` and `local-only` are indeed two different contracts and we don’t distinguish them explicitly. The LLM cascade (`nim-client.ts`) is OpenRouter → NIM → Ollama → raw query. “Local-only” (zero remote prompt-bearing attempts) is not a mode — it’s just what happens when OpenRouter and NIM are both unreachable. There’s no explicit `local-only` flag that guarantees zero remote egress. 5-7, 9-15: Architectural recommendations, all valid, none implemented. Will address in order of your priority list. **On the default route you proposed:** ```auto ``` ```auto benchmark v1 stays frozen ``` ``` ↓ ``` close the runtime contracts ``` ↓ ``` production-v1 really drives runtime (PARTIAL — 4/18 fields in ranker, rest in orchestration, unverified) local-only really blocks remote egress (OPEN — no explicit mode) live ranker == replay ranker (STRUCTURAL YES — same function; TEST MISSING — no direct parity assertion) query adaptation is observable (PARTIAL — data flags exist, not surfaced) health semantics survive into history (PARTIAL — archive, not state) ``` ↓ ``` then spend time on queryPlanner / fast mode / later ranking ideas The highest-information next step is #1 — verifying that all 18 `production-v1` fields are actually consumed end-to-end, not just declared. The ranker itself only reads 4; the rest belong to orchestration stages that need tracing. Will report back when the contract-closure pass is done. ---
**Author:** ![sookoothaii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sookoothaii/32/56632_2.png) [@sookoothaii](https://discuss.huggingface.co/u/sookoothaii)\ **Post date:** [September 8, 2026, 10:45pm UTC](https://discuss.huggingface.co/t/ghost-search-mcp-native-tor-routed-search-layer-for-rag-seeking-research-collaborators/179104/17 "2026-09-08T22:45:46Z")
**Contract-closure pass complete — all 5 points closed** The contract-closure pass is done. Every point from the original post is now closed with code, tests, and live verification. **#1 production-v1 really drives runtime — CLOSED** Profile reduced from 20 to 12 fields. Every remaining field drives runtime behavior end-to-end. The 8 removed fields were decorative — they offered mode names for which no code existed (`queryPolicy`, `rrfEnabled`, `trustWeighted`, `semanticDedupDefault`, `inputOrdering`, `qfCandidateDepth`, `duplicateRepresentative`, `outputLimitSemantics`). RRF and trust-weighting are invariants, documented in `rankProduction()` comments rather than exposed as toggleable flags. `profileToSearchParams(PRODUCTION_V1)` is now the merge base in the MCP server wrapper. The dashboard uses `PRODUCTION_V1` defaults instead of divergent hardcoded values — the `snippetFilter` divergence (dashboard `??true` vs profile `false`) is fixed. Commit `605a1ec`. 556 tests, 0 tsc errors. **#2 local-only really blocks remote egress — CLOSED** `GHOST_LLM_LOCAL_ONLY=1` activates an explicit local-only contract, distinct from local-fallback. In local-only mode, `activeConfig` returns only the Ollama fallback (local), and OpenRouter calls are guarded by `!this.localOnly`. NIM (primary) is never reached because `activeConfig` short-circuits to fallback when `localOnly=true`. Local-fallback = cascade behavior (OpenRouter → NIM → Ollama → raw) when remote endpoints are unreachable. Local-only = zero remote prompt-bearing egress, guaranteed by config, not by network failure. Three tests in `contract-closure.test.ts` verify the contract: default is cascade, `localOnly=true` activates local-only, and the two are distinct. **#3 live ranker == replay ranker — CLOSED** Structural parity was already guaranteed (same `rankProduction()` function imported by both paths). The missing direct-output parity assertion now exists: `parity.test.ts` (9 tests) covers REST/stdio semantic parity, and the live-vs-replay parity test was corrected to include engine-level info-first sorting. 556 tests pass. **#4 query adaptation is observable — CLOSED** `adaptQuery()` already computed `truncated`, `termsDropped`, `operatorsStripped` plus original/adapted length and term counts. The pipeline was discarding everything except `adapted.query`. Now: `EngineHealthEntry.adaptation` surfaces the full adaptation metadata per engine, in every path (ok, error, captcha, blocked, parse\_failure, exception). `SearchResponse.queryAdaptation` aggregates across engines: `enginesTruncated`, `enginesTermsDropped`, `enginesOperatorsStripped`, `anyAdaptation`. The audit log (`audit.ts`) writes `queryAdaptation` to stderr. The MCP server forwards it in the `ghost_search` response. Live verified: query with 14 terms and Boolean operators against 5 fast-mode engines. 4/5 engines dropped terms, 5/5 stripped operators, `anyAdaptation: true`. Per-engine adaptation differs correctly (torgle 14→6 terms, torch 14→14, no drop). Commit `3481ce6`. 584 tests (8 new), 0 tsc errors. **#5 health semantics survive into history — CLOSED** `health-persistence.ts` could save and load snapshots, but loaded data was only logged to stderr — never fed into breaker state. Snapshots were archive, not state. Now: `breaker.ts:restoreEngineHistory()` replays persisted events into the in-memory state, derives `successes`/`totalFailures`/`lastMs`/`lastCount`/`lastStatus`, truncates to `HEALTH_HISTORY_MAX` (50), preserves `deprecated` state, and does not open circuits from restored failures alone (restore is read-only history, not live failure). `health-persistence.ts:restoreHealthSnapshot()` iterates snapshot engines fail-soft. `server.ts` startup calls restore after load, logs `health_history_restored`. Live verified: saved snapshot (13 engines, 104 events), restarted server, all 13 engines restored with correct success/failure counts and `lastStatus`. Trend analysis (mean/median/p95 latency) identical before and after restart. No circuits opened from restored data. Commit `3481ce6`. 584 tests (20 new), 0 tsc errors. **Summary** | Point | Before | After | | --- | --- | --- | | #1 profile drives runtime | PARTIAL (4/18 in ranker, rest unverified) | CLOSED (12/12 wired) | | #2 local-only blocks remote | OPEN (no explicit mode) | CLOSED (GHOST\_LLM\_LOCAL\_ONLY + tests) | | #3 live == replay ranker | STRUCTURAL YES, TEST MISSING | CLOSED (parity test + sorting fix) | | #4 query adaptation observable | PARTIAL (data flags, not surfaced) | CLOSED (per-engine + aggregate + audit) | | #5 health survives restart | PARTIAL (archive, not state) | CLOSED (restore into breaker state) | 584 tests, 0 tsc errors. Moving to queryPlanner / fast mode / later ranking ideas per your priority list.