Benchmark and improve tool-search ranking with indexed BM25 #2996

Activity

1. sachinkahawala

   added

   enhancementNew feature or requestNew feature or request

   on Aug 2, 2026
2. github-actions

   added

   request ai review

   on Aug 2, 2026
3. CAOShurong commented on Aug 23, 2026

   

   CAOShurong

   on Aug 23, 2026

   More actions

   I independently prototyped the benchmark-only scope against current main (64a49f3) and think option 1 is the safest first step. It creates a measurable review surface without changing production ranking before the relevance labels themselves have been reviewed.

   The prototype uses 31 static, hand-labelled natural-language cases across 19 toolsets. It builds the real inventory through NewInventory(...).WithToolsets([]string{"all"}).Build(), then fails if a labelled tool is missing or belongs to a different toolset. On the current heuristic it measured:

   Metric	Result
   Recall@1	0.903
   Recall@3	0.968
   MRR@10	0.942

   Two implementation details seem worth preserving in an upstream harness:

   1. Validate labels against the current inventory before calculating metrics. That turns tool removal/renaming into an explicit benchmark-data failure instead of silently changing the denominator.
   2. Assert labelled target ranks and aggregate metrics across repeated inventory builds, not the complete ordering of irrelevant results. SearchTools currently has no explicit tie-breaker for equal scores, so requiring full result-list equality would encode an ordering that production does not promise.

   Local verification with Go 1.25.12 passed:

   go test ./pkg/tooldiscovery
    go vet ./pkg/tooldiscovery
    go test ./pkg/inventory
   

   A separate Go benchmark over the same 31 queries measured about 67.9 ms per full run, 26.7 MB, and 53,765 allocations on my Windows amd64 machine. I would keep latency/allocation reporting informational at first because those numbers are hardware-sensitive, while label validity, determinism, Recall@k, and MRR are appropriate regression checks after maintainers review the dataset.

   This dataset is smaller and independently labelled, so its absolute scores are not directly comparable to the 49-query prototype in the issue. The replication does support the proposed sequencing: land an auditable benchmark harness first, then evaluate BM25/hybrid changes in a separate patch against the accepted labels.
4. CAOShurong commented on Aug 23, 2026

   

   CAOShurong

   on Aug 23, 2026

   More actions

   Follow-up with a separate benchmark-only replication. This is intentionally not treated as the same dataset as the issue's 49-query/115-tool prototype, so the absolute numbers below are not directly comparable.

   Against the current main snapshot 64a49f3, I ran a second 49-case natural-language corpus over the complete registered inventory. The harness validates every labelled tool against the production constructor and checks toolset membership before scoring; repeated inventory builds also produced the same corpus keys and tool-name sequence.

   Observed current-heuristic results:

   Metric	Result
   Cases / deduplicated tools	49 / 117
   Recall@1	0.367
   Recall@3	0.571
   MRR@10	0.478

   Local verification on Windows amd64 (Intel Core Ultra 5 125H, Go 1.25.12) passed:

   go test ./pkg/tooldiscovery -run 'TestToolSearchBenchmark' -count=1 -v
   go vet ./pkg/tooldiscovery
   

   The benchmark run measured about 8.25–8.64 ms/query, 1.13 MB/op, and 2,739–2,740 allocs/op. I would keep those performance values informational because they are machine- and corpus-sensitive.

   The result reinforces the sequencing in the issue, but does not by itself justify changing production ranking: first agree on a canonical relevance-labelled corpus and inventory/snapshot semantics; then compare the current heuristic, BM25, and hybrid approaches against that corpus; only then consider a production change. I have not opened a PR or changed the ranking implementation.