VIBE¶

The target is the CPU overview on VIBE's modern embeddings, using its published default dataset selection and recall@100. SIFT and GloVe results do not establish that WARM is competitive here. Report the complete recall/throughput curve, per-dataset results and target coverage alongside the overview score.

Published comparison, 2026-10-21¶

The saved published inputs and recomputed ranking reproduce the live overview. The score averages each method's QPS divided by the fastest eligible CPU QPS on each dataset. An unreached recall target contributes zero; it is not dropped from the denominator. The snapshot contains k=100 results only.

These are measured points, not inferred crossover boundaries. At 1.00, PyNNDescent reaches the target on 10/11 datasets, NGT-QG on 6/11, PAG on 4/11, and Glass on 3/11. A first-place average does not mean first place everywhere. The result JSON retains the per-dataset comparisons.

VIBE's exporter counts returned distances within 0.001 of the exact kth-neighbor distance. Consequently, its 1.00 recall is not a guarantee of exact neighbor-ID recovery. Local diagnostics record both metrics, without rounding a near-1 value up to 1.

Large raw predictions, full timing reports and the deployed Parquet snapshot use Git LFS. Fetch them before replaying or checking the saved results:

git lfs pull --include='benchmarks/results/vibe_v1/**'

This is the published circle's selection, not every dataset in the upstream repository. The additional 2048-dimensional iNaturalist set is a separate dimension stress test. Larger-dimensional additional sets must not silently replace the default circle.

Measured risk¶

The checked diagnostic confirms a dataset-dependent loss with the current 4-bit configuration. These results use 100,000 indexed rows and 256 original queries, not the full datasets. The follow-up below identifies a graph connectivity problem and tests an encoding change; the initial deficit is not an established 4-bit ceiling.

The two Glass rows demonstrate why one arbitrary quantization configuration cannot represent that method. These are fixed-build diagnostic sweeps, not the complete author parameter grids. All six completed adapter results were checked against their saved predictions and timings, with independent float64 oracle spot checks against the original vectors.

A native ablation then holds the graph, queries and search configurations constant while changing traversal precision. It expands search up to ef=2000 and rerank=2000 and measures strict neighbor-ID recall, separately from the tolerant VIBE metric above:

Each pair has an identical recorded graph hash and identical search grids. Both retain an 8-bit residual. The larger loss occurs in the tested text-to-image workload, while the two higher-dimensional in-distribution sets retain high recall. This does not establish behavior on the other VIBE datasets or at their full scale. The wider native 4-bit sweep reports strict recall; its tolerant VIBE recall was not measured.

LoRANN's trial exits with SIGILL after building; the cause is unestablished. PyNNDescent's trial returns duplicate IDs and fails validation. Their failure logs are retained. Neither trial supplies a valid score or invalidates its method's published results. NGT-QG's native CLI is not installed. PAG remains blocked on an AVX-512 host.

No throughput winner is claimed from these initial diagnostic runs: they were not interleaved and some builds overlapped. The native ablations also exposed an existing Linux load-reporting bug: the driver tried macOS sysctl, leaving load unknown. That bug is fixed for subsequent runs; the saved native results retain unknown load and failed timing gates.

Measured improvement¶

The independently checked follow-up compares four encodings with the same graph, original queries and search grid. An existing three-round signed Walsh-Hadamard transform, fixed seed 24301, raises LAION's maximum VIBE recall from 0.933555 to 0.980469 with 4-bit traversal. The transform adds 6,144 bytes at 512 dimensions. Query transformation is included in timing. All configurations retain an 8-bit residual.

The shared 0.980469 ceiling in rotated 4-bit and both 8-bit arms led to a second control. The cached directed graph has 63 strongly connected components, with 99,932 vertices in the largest one. Four small sink components have no outgoing path. A deterministic, data-only control adds one edge from each sink to the nearest vector in the largest component. It uses no queries or neighbor truth, preserves every existing edge, and fits the existing degree limit and index allocation. The four edges and input hashes are recorded; the original cache is preserved.

This separates two effects: quantized entry selection can lead into a graph region that cannot be escaped, and rotation improves the accuracy of the compact codes. The earlier 4/8-bit difference therefore does not establish a recall ceiling intrinsic to 4-bit storage or out-of-distribution queries.

On the graph with escape edges, the fastest measured configurations reaching 0.99 VIBE recall are:

Rotated 4-bit is 1.76x plain 4-bit by these medians; the three paired-round ratios are 1.64x, 1.77x and 1.72x. It uses 20.8% less index storage than rotated 8-bit. At 0.95 recall, 8-bit is faster: plain 8-bit reaches 4,002 QPS, versus 2,612 for rotated 4-bit. At 1.00 recall, rotated 4-bit reaches 549 QPS at ef=2000/rerank=400 with strict ID recall 0.996797. The faster curve varies with the target; rounding strict recall to 1.00 would be incorrect.

The original-graph controls also show why rotation should remain a measured choice. On arxiv-768 it pads to 1024 dimensions: 4-bit index storage grows from 135,983,700 to 174,397,012 bytes and QPS at 1.00 VIBE recall falls from 2,092 to 1,629. On the additional iNaturalist-2048 stress test, it adds only 24,576 bytes and QPS at 1.00 VIBE recall rises from 389 to 1,147. These are the fastest eligible points on the fixed grid, not extrapolated speedups or new public leaderboard scores.

All four reports retain raw original IDs and per-query nanoseconds. The checker recomputes every reported recall, timing and target selection and checks four queries against all original corpus vectors. Missing neighbors receive zero credit. Timings alternate the four encodings within each of three rounds on one pinned CPU, with no concurrent benchmark builds. Runs on the two graph versions are separate; their timing differences are not a paired graph-speed comparison. The source and binary receipt records exact versions. The controls remain experimental: the VIBE adapter and production construction defaults have not adopted either change, and full-scale default-circle coverage remains unmeasured. All tested graphs use pruning alpha=1.0; this does not establish the optimum construction setting or a general connectivity repair for production.

Measurement protocol¶

The first diagnostic uses pinned VIBE files at dataset revision 07b387891a221b7b073b83d2f752b76462e5fa03: 100,000 corpus rows and 256 original test queries for arxiv-nomic-768, laion-clip-512 and inaturalist-resnet-2048. The corpus is a prefix. Ground truth is recomputed over that prefix; the full-corpus neighbor IDs are never reused. LAION queries retain the original text-to-image distribution. This is a bounded diagnostic, not an official-scale result or a held-out final evaluation.

Each method uses its VIBE adapter and one explicitly recorded build, with the query grid fixed before evaluation. This is not the complete build-parameter search used for the published leaderboard. Adapter builds and batch-one search run on one pinned Linux CPU. Native arxiv/iNaturalist graph builds use four threads, recorded in their reports; native query loops use one thread. Three query timing repeats, raw adapter IDs and per-query timings, source hashes, package versions, load, build time and process memory are saved. Targets 0.73, 0.77, 0.90, 0.94, 0.95, 0.99 and 1.00 use the fastest measured configuration reaching the target; a missing target stays missing. Timing comparisons remain exploratory: methods are not interleaved and this machine differs from VIBE's hardware. Do not compare these QPS directly with public QPS.

The accepted WARM implementation is commit 28a0a5b1bc6e84037fed51fdba8c6b508f821502: 4-bit mean-centered traversal codes, 8-bit residual reranking, aligned rows, BFS order, 3-byte IDs and quantized graph construction. The default wrapper does not rotate traversal vectors; the earlier wrapper description was incorrect. A rotation or 8-bit ablation is a separate experiment and must be identified as such.

The benchmark host is an Intel Core Ultra 9 185H under WSL2 with AVX2 and no AVX-512. PAG's pinned release ee34ed78b7a28225b2db6210ad942152e1cd540d requires its AVX-512 backend. An AVX-512 host is required for an honest direct PAG comparison. The WARM Python adapter serves all four VIBE float distances: cosine and normalized by angle, euclidean directly, and ip through the exact inner-product-to-Euclidean reduction (one appended coordinate per row).

PAG requires AVX-512, which this host lacks, so it stays unmeasured here.

The completed local experiment uses encoding-only rotation on each cached graph, using the existing three-round signed Walsh-Hadamard transform with fixed seed 24301. The experimental diagnostic compares plain/rotated 4-bit and 8-bit encodings with an 8-bit residual, identical graph topology, ef 128/256/512/1000/2000 and rerank 100/400/1000. Query transformation is timed. The arms alternate in each timing round, and original-query distances plus strict IDs are both scored. Rotation pads to a power of two; the resulting code bytes and transform state must be counted, particularly at 768 dimensions. Production defaults remain unchanged. Its reproducible commands, after building the pinned graph cache and preparing the native data, are:

cargo build --release --locked --features experimental,cli --bin vibe-rotation
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 RAYON_NUM_THREADS=1 \
  taskset -c 0 target/release/vibe-rotation --data /path/to/native-laion \
  --graph-cache /path/to/laion.graph --out /path/to/rotation-laion.json
uv run --no-project --with numpy==2.2.6 --with scipy==1.17.1 --with h5py==3.15.1 \
  python benchmarks/vibe/connectivity.py /path/to/laion.graph \
  --data /path/to/native-laion --escape-cache /path/to/laion-escape.graph \
  --out /path/to/laion-escape-control.json
# Repeat vibe-rotation with the escape cache and a new output path, then check:
uv run --no-project --with numpy==2.2.6 --with h5py==3.15.1 \
  python benchmarks/vibe/check_rotation.py /path/to/rotation-laion.json \
  /path/to/rotation-laion-escape.json --data /path/to/hdf5-data \
  --out /path/to/rotation-validation.json

Use to_native.py to create a fresh native directory containing distances.f32 as well as vectors and IDs. The original diagnostic native directories predate that additional file and remain unchanged. The follow-up adds 14 Python integrity/connectivity regressions; the qgraph test run passes 36 tests and retains its three explicitly ignored experiments.

Contribution¶

The WARM adapter follows VIBE's new-method guide: one-thread builds, unique original row IDs, separate build/query parameters, four build combinations and 45 query combinations per metric, with a pinned implementation revision in image.def. It is the quantized graph algorithm, not the memory database service. It advertises cosine, normalized, euclidean and ip datasets; ip is served directly by inner product on the original coordinates (direct_ip). The grid and build flags are fixed below.

Reproduce the published ranking and focused regression checks:

uv run benchmarks/vibe/leaderboard.py benchmarks/results/vibe_v1/leaderboard_inputs \
  --out /path/to/leaderboard.json
uv run --no-project --with numpy==2.2.6 --with h5py==3.15.1 \
  --with faiss-cpu==1.14.3 --with fsspec --with aiohttp --with duckdb==2.0.0 \
  --with pytest python -m pytest -q benchmarks/vibe/test_measurement.py

For the diagnostic, install the pinned WARM wheel and comparator dependencies in an isolated environment, copy benchmarks/vibe/warm to vibe/algorithms/warm in VIBE revision 7ac76c1f8320a411ce7c9cb3791bdd7f2349b30c, then run:

uv run --no-project python benchmarks/vibe/prepare.py laion-clip-512-normalized \
  --out /path/to/data
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMBA_NUM_THREADS=1 \
  uv run --no-project python benchmarks/vibe/run.py --vibe /path/to/vibe \
  --data /path/to/data/laion-clip-512-normalized-n100000.hdf5 \
  --method warm --out /path/to/laion-warm.json

An upstream contribution also needs the WARM row in VIBE's algorithm table and a Linux image build through ./install.sh --algorithm warm. The prepared upstream patch includes the README row and all three adapter files; it passes the reverse-apply check on the pinned checkout (VIBE 7ac76c1). The image has not been built: this host has no Apptainer, so the same recipe was replayed in a micromamba copy of base.sif and run.py --local passed on a smoke subset of llama-128-ip. The real adapter checks cover both advertised metrics, dimensions 768/2048, original IDs, input immutability, result-count boundaries and VIBE's configuration loader. Local native-wheel validation does not establish that the Apptainer image has built successfully.

The native driver now supports ann-qgraph --count 100 as well as its existing default of 10 and emits the corresponding recall field. Seven executable regressions cover corpus/truth boundaries, preservation of recall@10, empty queries and Linux load reporting. Nine diagnostic regression tests pass; public-bench also passes the scoped all-features check and Clippy with warnings denied. Build/check logs and the source/host receipt are retained.

Image build flags¶

image.def compiles the wheel with -C target-cpu=x86-64-v3 on x86_64 (the default target elsewhere) instead of target-cpu=native. VIBE's install.sh builds the .sif on whichever host runs it (its Slurm example uses a scratch node) and the image is then reused on the run nodes: Intel Xeon Gold 6230 (Cascade Lake, AVX-512F/BW/VL/VNNI, no VEX AVX-VNNI) for the current results and AMD EPYC 9965 (Zen 5) for the next refresh. A native build on a newer build host can emit instructions that fault on the run node. Every x86 qgraph kernel is selected at run time (is_x86_feature_detected: AVX2, AVX-VNNI, AVX-512F/VL, AVX-512 VNNI) and the pinned revision has no compile-time x86 target_feature gate, so the AVX-512 VNNI kernels still run on both node types; the baseline only sets the code generation of the non-kernel code. x86-64-v4 was not chosen because it faults on run hosts without AVX-512. The Modal runner (benchmarks/vibe/modal_vibe.py) keeps target-cpu=native because it compiles in the container that runs the job.

PGO development verification, 2026-10-24 UTC: Intel Core Ultra 9 185H (no AVX-512), older WARM source 99e84d92f72a42e6f54b2cc48fb30fc86cdfa99c, three existing binaries (x86-64-v3, native, and x86-64-v3 + PGO), and the same saved indexes for all arms. The profile used nytimes-256-angular training data (2,000 holdout queries requested, 1,997 evaluated); glove and sift were evaluation datasets. The fixed protocol used ann-qgraph --index vertex --residual --align-rows --reorder --quantized-search --degree 32 --build-ef 200, k=10, 10,000 queries, ef 64/128/256, rerank 100, one thread pinned to CPU 0, and three rotated arm orders. The whole repeat held the timing lock for 70 seconds; maximum recorded 1-minute load was 0.70, below the strict 3.0 gate. Recall and mean distance-evaluation counts matched across arms. Median paired QPS ratios, rounded to six decimals:

PGO won all 18 pairs and cleared the pre-registered 3% median development bar at all six points. Native lost all nine glove pairs and four of nine sift pairs; the v3 portability choice is unchanged. All raw measurements, paired ratios, hashes, lock receipt, and the earlier contended run (including its two PGO losses) are preserved in vibe_pgo_verification_v1. This verifies only the older binaries on these k=10 workloads. The current engine, wheel profile training, and image build remain unvalidated, so image.def does not adopt PGO. This result does not establish VIBE k=100 performance or a leaderboard win.

Pre-registered OOD grid, 2026-10-23¶

Fixed in benchmarks/vibe/warm/config.yml before any run of this grid. Build arguments degree x build_ef x code_bits x query_aware, query arguments ef x rerank:

Both are within VIBE's guidance (build combinations below 20, query combinations below 60).

• OOD sets (VIBE README, "Out-of-distribution datasets"): hotpotqa-harrier-640, imagenet-align-640, laion-clip-512 (normalized), yandex-200 (cosine), yi-128-ip, llama-128-ip, cqadupstack-lemur-2048-ip, cqadupstack-muvera-5120-ip (ip). Every entry sets ood: true, so VIBE's runner calls fit_ood with the dataset's official learn split; the wrapper samples 32768 learn rows (numpy default_rng(24301) permutation, all rows when the split is smaller) and never reads the learn neighbors, test queries or ground truth. The run name records learn_queries=N.
• VIBE grids are per metric section, not per dataset: the in-distribution sets of the cosine, normalized and euclidean sections (no learn split) run the same grid through fit, the plain graph. Only the ip section is entirely OOD.
• ef reaches down to 40: the query-aware graph halved the ef needed on coco (100k slice, recall at ef 20 0.852 -> 0.935), so the low end of the curve moves left; the upper end is unchanged at 1000 for the in-distribution sets.
• ip uses direct inner-product serving (direct_ip); the previous query-aware ip runs covered ef 100..1000 with rerank 400, so this grid brackets that range (64..2000) with rerank 200/400/1000. The earlier ef up to 8000 came from serving ip through the lifted Euclidean reduction, which this wrapper no longer does.