What operations aren't optimal?
For small corpus sizes, the 0.1% pass rate threshold causes searches to operate many multiples slower than they could. I was testing agent memory using LongMemEval which has 500 "haystacks" in the same HNSW index making up a total corpus of ~124k vectors. The pass rate of a query on a specific haystack is ~0.2%. This is higher than our 0.1% threshold and inline filtering would be used, which was 4-8x slower than pre-filtering depending on the configuration I used.
How could they be improved?
I had some testing done to get a rough idea of the inflection point based on the corpus size across several different hybrid queries for both throughput and latency: https://github.com/BCathcart/valkey-search/blob/brennan-filtering-decision-investigation/PREFILTER-REPORT.md. It's clear that the threshold percentage is a function of corpus size and needs to be varied.
I asked Fable for a formula as an example:
Formula
UsePreFiltering ⟺ |F| ≤ C · √(N · k/10) (default C = 3)
where |F| = estimated filter match count, N = vectors in the index, k = KNN k.
Summary
Benchmarks across 5k–1.28M vectors and 7 query shapes show the inline/prefilter crossover is not a fixed fraction of the corpus: expressed as a percentage it falls from ~8% at N=5k to ~0.4% at N=1.28M. Expressed as an absolute match count, the crossover follows |F|* ≈ C·√N (measured exponent 0.55), with C between ~3 (tag, composed AND filters) and ~7 (single numeric) across both latency and throughput.
This matches the cost model: prefilter cost is brute-force KNN, linear in |F|; inline cost is HNSW over-expansion to find k passing nodes, proportional to k·N/|F|. Equating the two gives |F|* ∝ √(k·N), which also predicts the threshold scales with √k (benchmarks used k=10; the √(k/10) term is extrapolated, not measured).
We pick C=3, the floor of the observed crossover range, because misestimation costs are asymmetric: wrongly choosing inline costs a bounded few ms, while wrongly choosing prefilter grows linearly with |F| (up to ~100x at 10% selectivity on 1.28M). The current fixed-ratio default (0.001·N) is 3–40x too conservative at all measured sizes and over-aggressive when extrapolated beyond ~10M.
I'm sure if we ran some very extensive benchmarks and passed the results to AI , it could come up with a pretty good formula. If we want to maintain precision though, we'll need to continuously update it as our performance characteristics are rapidly being tweaked and tuned right now.
Another thing to consider is that favouring pre-filtering should help recall.
What operations aren't optimal?
For small corpus sizes, the 0.1% pass rate threshold causes searches to operate many multiples slower than they could. I was testing agent memory using LongMemEval which has 500 "haystacks" in the same HNSW index making up a total corpus of ~124k vectors. The pass rate of a query on a specific haystack is ~0.2%. This is higher than our 0.1% threshold and inline filtering would be used, which was 4-8x slower than pre-filtering depending on the configuration I used.
How could they be improved?
I had some testing done to get a rough idea of the inflection point based on the corpus size across several different hybrid queries for both throughput and latency: https://github.com/BCathcart/valkey-search/blob/brennan-filtering-decision-investigation/PREFILTER-REPORT.md. It's clear that the threshold percentage is a function of corpus size and needs to be varied.
I asked Fable for a formula as an example:
I'm sure if we ran some very extensive benchmarks and passed the results to AI , it could come up with a pretty good formula. If we want to maintain precision though, we'll need to continuously update it as our performance characteristics are rapidly being tweaked and tuned right now.
Another thing to consider is that favouring pre-filtering should help recall.