Gaps: where WARM loses or ties¶
Generated with the table from the committed results; a gap stays here until a measured change closes it. Factor is WARM advantage (below 1 = WARM worse). Hypotheses are unmeasured unless a file is cited.
| Status | Dataset | Hardware class | Metric | Comparator | WARM | Comparator value | Factor [interval] | Hypothesis | Code path |
|---|---|---|---|---|---|---|---|---|---|
| LOSS | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | Ingest/build time | SQLite FTS5 | 2.515 s | 2.061 | 0.82x [0.79, 0.85] | unprofiled hypothesis: per-document text encoding in documents::prepare (src/core/encoding.rs) dominates ingest; profile it, then a lexical-only ingest path or a cheaper encoder is the lever | src/core/documents.rs prepare + src/core/engine/document_api.rs memorize_documents |
| LOSS | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | Peak RSS, build | SQLite FTS5 | 132.8 MiB | 120.5 | 0.91x [0.91, 0.91] | unprofiled hypothesis: the arena and index are built in memory before commit; streaming commits per batch | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | Peak RSS, serving | SQLite FTS5 | 65.91 MiB | 25.83 | 0.39x [0.38, 0.41] | the whole store is resident after open; memory-mapped postings and file-backed text (ledger 2A/2B) are the open levers | src/core/storage.rs (arena), src/core/documents/index.rs, postings.rs |
| LOSS | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | Cold start to first answer | tantivy | 0.6196 s | 0.07182 | 0.12x [0.10, 0.13] | the store is read into memory at open (open_s in the raw file); lazy or memory-mapped loading of the arena and postings is the lever | src/core/storage.rs and src/core/engine/mod.rs (open: arena and document map load) |
| TIE | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | nDCG@10 | Apache Lucene 9.12.3 (JVM) | 0.4561 nDCG | 0.4627 | 0.99x [0.60, 1.65] | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs | |
| TIE | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | recall@100 | SQLite FTS5 | 0.8203 recall | 0.8392 | 0.98x [0.79, 1.22] | none: a rounding-level difference in analyzer details (stemmer, stop words), not a lever | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| TIE | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | QPS (single thread) | tantivy | 2289 QPS | 2504 | 0.91x [0.83, 1.04] | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs | |
| TIE | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | p50 latency | tantivy | 420.2 us | 379.6 | 0.90x [0.83, 1.04] | none identified; the median is within noise of the leader | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| TIE | memory (4,559 documents, 100 queries) | modal-x86_64-8vcpu-32g | p99 latency | tantivy | 775 us | 720.4 | 0.93x [0.86, 1.02] | tail queries are the long ones; MaxScore-style early termination for long queries (the ledger's impact-index work, category 1C) is the candidate | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | Ingest/build time | SQLite FTS5 | 0.5922 s | 0.2858 | 0.48x [0.47, 0.49] | unprofiled hypothesis: per-document text encoding in documents::prepare (src/core/encoding.rs) dominates ingest; profile it, then a lexical-only ingest path or a cheaper encoder is the lever | src/core/documents.rs prepare + src/core/engine/document_api.rs memorize_documents |
| LOSS | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | Peak RSS, build | SQLite FTS5 | 57.3 MiB | 33.82 | 0.59x [0.58, 0.60] | unprofiled hypothesis: the arena and index are built in memory before commit; streaming commits per batch | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | Peak RSS, serving | SQLite FTS5 | 56.79 MiB | 29.93 | 0.53x [0.50, 0.54] | the whole store is resident after open; memory-mapped postings and file-backed text (ledger 2A/2B) are the open levers | src/core/storage.rs (arena), src/core/documents/index.rs, postings.rs |
| LOSS | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | On-disk size | Apache Lucene 9.12.3 (JVM) | 2.063 MiB | 1.487 | 0.72x (no CI) | compact postings and file-backed text (ledger 2B/2C) and removing preallocated arena slack | src/core/storage.rs (arena preallocation), src/core/documents/postings.rs |
| LOSS | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | Cold start to first answer | tantivy | 0.1754 s | 0.06822 | 0.39x [0.35, 0.43] | the store is read into memory at open (open_s in the raw file); lazy or memory-mapped loading of the arena and postings is the lever | src/core/storage.rs and src/core/engine/mod.rs (open: arena and document map load) |
| TIE | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | nDCG@10 | SQLite FTS5 | 0.2916 nDCG | 0.2914 | 1.00x [0.70, 1.42] | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs | |
| TIE | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | recall@100 | Apache Lucene 9.12.3 (JVM) | 0.2274 recall | 0.2276 | 1.00x [0.69, 1.45] | none: a rounding-level difference in analyzer details (stemmer, stop words), not a lever | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| TIE | beir:nfcorpus (3,633 documents, 324 queries) | modal-x86_64-8vcpu-32g | p99 latency | tantivy | 279.1 us | 293.7 | 1.05x [0.84, 1.21] | tail queries are the long ones; MaxScore-style early termination for long queries (the ledger's impact-index work, category 1C) is the candidate | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | QPS (single thread) | tantivy | 2733 QPS | 2965 | 0.92x [0.87, 0.98] | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs | |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | p50 latency | tantivy | 372.7 us | 334 | 0.90x [0.85, 0.96] | none identified; the median is within noise of the leader | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | Ingest/build time | SQLite FTS5 | 1.419 s | 0.4272 | 0.30x [0.28, 0.32] | unprofiled hypothesis: per-document text encoding in documents::prepare (src/core/encoding.rs) dominates ingest; profile it, then a lexical-only ingest path or a cheaper encoder is the lever | src/core/documents.rs prepare + src/core/engine/document_api.rs memorize_documents |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | Peak RSS, build | SQLite FTS5 | 65.45 MiB | 39.21 | 0.60x [0.59, 0.61] | unprofiled hypothesis: the arena and index are built in memory before commit; streaming commits per batch | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | Peak RSS, serving | SQLite FTS5 | 159.7 MiB | 54.49 | 0.34x [0.34, 0.35] | the whole store is resident after open; memory-mapped postings and file-backed text (ledger 2A/2B) are the open levers | src/core/storage.rs (arena), src/core/documents/index.rs, postings.rs |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | On-disk size | Apache Lucene 9.12.3 (JVM) | 2.91 MiB | 1.987 | 0.68x (no CI) | compact postings and file-backed text (ledger 2B/2C) and removing preallocated arena slack | src/core/storage.rs (arena preallocation), src/core/documents/postings.rs |
| LOSS | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | Cold start to first answer | SQLite FTS5 | 0.2393 s | 0.07661 | 0.32x [0.27, 0.37] | the store is read into memory at open (open_s in the raw file); lazy or memory-mapped loading of the arena and postings is the lever | src/core/storage.rs and src/core/engine/mod.rs (open: arena and document map load) |
| TIE | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | nDCG@10 | Apache Lucene 9.12.3 (JVM) | 0.7027 nDCG | 0.7022 | 1.00x [0.90, 1.11] | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs | |
| TIE | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | recall@100 | SQLite FTS5 | 0.9264 recall | 0.9322 | 0.99x [0.94, 1.05] | none: a rounding-level difference in analyzer details (stemmer, stop words), not a lever | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |
| TIE | beir:scifact (5,183 documents, 809 queries) | modal-x86_64-8vcpu-32g | p99 latency | tantivy | 518.3 us | 508.3 | 0.98x [0.87, 1.19] | tail queries are the long ones; MaxScore-style early termination for long queries (the ledger's impact-index work, category 1C) is the candidate | src/core/documents.rs (prepare, search), src/core/documents/search.rs, src/core/engine/document_api.rs |