Feature tour

What the engine can be asked, and what it does with each request. Every feature on this page follows the same two rules that the rest of the design does: the answer the fleet returns is the answer a single machine holding the whole corpus would return, and a request the engine cannot answer that way comes back as an error that names the reason.

flowchart LR
        req["a request"] --> sel["1. Selection
which documents qualify
and their starting score"]
        sel -->|"a fixed pool of candidates"| boost["2. Rescoring
extra queries over that pool"]
        boost --> score["3. Scoring
combine the named numbers into one"]
        score --> shape["4. Shaping
sort, group, page, count, explain"]
        shape --> res["results, each carrying
the numbers behind its score"]

Query shapes and hybrid fusion

The first stage of a query is described by a small tree. Its parts are of four kinds. A scoring leaf asks a real question and hands back a number for each document it matched, and there are two of those: a keyword question scored by BM25, and a vector question scored by similarity to the query's own vector. A filter leaf decides membership and returns no score at all. A composite node says how the scoring leaves should be combined into one ranking. And a boolean node holds other nodes underneath it with the familiar words: must, should, must_not, and a threshold saying how many of the shoulds have to hold.

Membership and scoring are kept apart on purpose. Saying that a document has to match both a keyword question and a vector question is a statement about which documents qualify. It says no more than that. How the two relevance numbers should be turned into one ordering is a separate question, and there are several reasonable ways to answer it. Those ways are the composite strategies, and they are the subject of the widget below.

Two of the strategies deserve a word first, because they are the ones with unfamiliar names. Reciprocal rank fusion throws the scores away and keeps only the positions: a document that came first in one ranking and fourth in the other is scored from those two positions alone, which makes it safe to combine two rankings whose numbers are on completely different scales. A blend does the opposite. It rescales each ranking's scores onto a common range and then averages them, so a document that won its ranking by a wide margin keeps that margin.

Pick a strategy below. The list is the same six documents with the same two scores each, ordered the way that strategy orders them.

Two more settings sit on the combined route. Giving one of the two questions a weight of zero turns it off, which is how a vector-only or keyword-only query runs through the same code as a combined one. And min_vector_score requires every returned result to clear a minimum vector score, applied before the two rankings are combined and before the list is cut to length, so a qualifying document further down is promoted instead of the page simply getting shorter.

Filters

A condition on a query is written in CEL, a small expression language from Google that reads like the boolean part of a programming language and is designed to be evaluated safely and quickly on data it did not write. The same language is used everywhere the engine needs a short expression: filtering, computing a value to sort or aggregate by, and shaping what comes back. The coordinator parses and compiles the expression once into a typed tree of comparisons; the shards never see the text. Each shard then resolves that tree against its own vocabulary, turning column names into table positions and quoted words into the numbers its dictionary uses for them, so what runs on the hot path is integer comparison.

court == "scotus" && year >= 1990 && "color" in tags
cites["410 U.S. 113"] >= 3 || within_radius(courthouse, 38.9, -77.0, 25000.0)
decided >= timestamp("2015-01-01T00:00:00Z") && !(court in ["ca5", "ca9"])

docs/cel-filters.md, the selection language

The column plane a filter reads is typed: facet columns, 64-bit integers, doubles, maps, timestamps, and geographic points. On the vector side the resolved condition becomes a list of which rows are still allowed, handed to the scan as a mask, so a document the condition removed is never scored. Both halves of a combined query take the same tree, which is why a combined query carrying a condition describes the set of documents it actually returned. A column no shard declares comes back as an error naming the column, so a typo is not silently an empty result.

Sort and collapse

Sort orders by columns instead of relevance: most significant key first, ties broken by the next key, then by document id. Keys can be integer, double, or facet columns, or the lineage keys that tie a chunk to its parent document. A document with no value for a key is excluded, the same stance the filters take, because absence has no position in a column order. Numbers travel between shard and coordinator as order-preserving bits so the merge is a byte comparison.

Collapse returns one representative per key value with the group's own top hits beside it, plus how many hits the group had in the candidate pool and whether that count is known to be complete. A single leaf's ordering does not change when you look deeper into it, so the coordinator can keep doubling the pool until the page has enough groups. An ordering that does change with depth, which is what a fusion strategy or a rescoring step produces, is not deepened without being asked: a pool too short to fill the page comes back as an error naming the setting to raise.

Facets and aggregations

Facets are the counts that sit beside a result list telling you how the whole set of matches breaks down: 4,102 from this court, 380 from that one. This engine counts them over every document that matched at least one of the query's words, not over the handful that survived onto the page. Counting only the survivors is cheaper, and it is wrong in a way readers notice on a broad query, because the sidebar then describes the page rather than the search. The cost was priced before the feature was adopted: the union walk runs at about 1.2 ns per posting and the ordinal counting at about 2 ns per matched document, which on a shard of roughly 10.8 million documents puts a stopword-heavy worst case at 15 to 35 ms per shard, in parallel across the fleet, and at zero when a request asks for no facets.

The Aggregate route folds up to 32 named expressions over the filter-admitted set: count, sum, min, max, mean, variance, standard deviation, exact percentiles, and exact distinct-value cardinality. Every one of them is exact and repeatable to the bit. Integer sums accumulate in 128-bit integers on the shard and merge in 128-bit integers at the coordinator; a total that leaves the 64-bit range comes back as an error naming double() as the fix. Percentiles are nearest-rank order statistics found by a count-below binary search, not a sketch. Cardinality unions the actual distinct sets and carries a cap, so a request that would union a million values comes back as an error naming the aggregation and the two ways out, instead of handing back an estimate.

Histograms come in two forms. A fixed-interval histogram buckets any numeric expression. A calendar histogram names a unit (minute, hour, day, ISO week, month, quarter, year) and buckets an integer expression in epoch microseconds at civil boundaries, in a fixed UTC offset. The calendar is proleptic Gregorian computed in exact integer arithmetic, so leap years and month lengths are what the civil calendar says, dates before 1970 included. Named time zones with daylight-saving rules are not implemented.

The same request can also ride along on a search. QueryRequest.aggregate folds over the candidate pool the page was drawn from, which fixes the pool depth so that page two reports the same counts as page one, to the bit.

Explain

Setting explain on a request hands every hit a tree of the arithmetic that produced its score. Each node's description states how its value follows from its children, and the root's value is the score that was served. No number in the tree is recomputed: they are the numbers the engine used on the path that produced the hit, assembled afterwards, which is what makes "the same hits, bit for bit, with the flag on" a test rather than a claim. A keyword leaf reports one contribution per field and word, with the numbers that produced it underneath: how often the word occurs in the document, how many documents contain it, how unusual that makes it, and the document's length against the average. A composite reports one node per ranking, with the formula that combined them. A boolean root reports the sum of its positive clauses. Three shapes compute no score and return an error instead of a tree: a filter-only browse, a lexical leaf being walked for a column sort, and the streaming route, whose provisional revisions would explain a score that was never served. Real trees, from a live engine, are on the demo page.

Text features

Analysis is either the in-repository Rust analyzer, which runs in-process on servers and native clients, or the OpenNLP sidecar over gRPC, which adds sentence detection, model analysis, and static embeddings. The query is analyzed with the same specification the documents were, and a mismatch is a request failure instead of a silently empty result set.

Synonyms are a query-time rewrite. A rule is either symmetric (every entry expands to every other) or one-way. Rules live in the coordinator's table and can also arrive on the request. Entries are written as surface words and analyzed under the field's own chain, stemmer included, so a rule written as automobile matches the stem the dictionary holds. No term is added at ingest and no posting is written.

Did-you-mean and autocomplete both read the term dictionary, which is already stored in byte order. Suggest completes a prefix against one indexed field ranked by document frequency summed across the shards, which is exactly the single-index answer. It is bounded on purpose: the limit defaults to 10 and a limit above 100 comes back as an error, and a prefix with more matching dictionary terms than max_scan allows comes back as an error naming the count, instead of a truncated list that looks complete.

Phrases and proximity. A phrase match requires its analyzed terms in order inside a window that allows at most slop extra token positions, and a slop of zero is an exact phrase. The positions are token ordinals from the ingest analysis, never character offsets, because a dropped token and a run of whitespace look the same between two character spans and a phrase index built on that guess would match "new york" against "new‑york" and call it exact. A field that stored neither payload comes back as an error naming itself.

Highlighting is done on the shard. The analyzer's sentence spans are stored at ingest, and a snippet is cut from the stored text against those spans, so a reader gets the sentence a term was found in. Offsets are UTF-16 units of the original text, before any accent folding or case folding touched it, and no analyzer runs on the query path.

Relevance tuning

Score stages are the tuning that runs during the search: recency decay, a level boost, a citation weight, geographic decay. A stage is a pair, not a function: an evaluation rule and a bound rule. The bound is what lets block-max pruning stay sound while scores change under it, and a stage the engine cannot lift a safe upper bound through is not admitted. That is why an arbitrary expression does no scoring here: CEL selects, and a fixed vocabulary of bounded stages scores.

The composite scorer is the learning-to-rank surface. It runs on the coordinator over a fixed candidate pool, after selection and after boost signals are computed. Each dimension names its source (the composite base score, a search leaf's raw relevance by id, a boost query's relevance by id, or a bounded stored-value function), its weight, its normalization, and what to do when the signal is missing. The combination is a weighted sum, weighted mean, maximum, product, geometric mean, or harmonic mean. The response reports the normalized value and the weighted contribution for every dimension, and the test that keeps it usable is that a client can recompute every final score from the response alone.

A/B comparison is a first-class request. VariantSearch takes two or more whole query configurations, runs them over one corpus at one shared depth, and returns each ranking plus rank-difference measures against the first arm: overlap, rank-biased overlap, and an asymmetric score regret read from the reference arm alone. A variant carries a whole request rather than a set of overrides, so a lexical-only arm can be compared against a combined one, which is how a question like "is the vector half worth the time it costs" gets asked. With exactly two arms it can also return a team-draft interleaving, merged and ready to serve, with a seed derived from the query text so a re-run reproduces the same merge. These measures report disagreement, not quality; which arm is better needs labels or the interleaving's clicks.

Collections

One cluster can serve several datasets, and the isolation is structural rather than a name check. A collection owns its own shard set, topology, statistics cache, calibration, analysis backend, BM25 parameters, phrase index, and control plane. There is no table two collections share, so the same term has a different document frequency in each because each keeps its own statistics over its own shards. A shard belongs to exactly one collection, reports it in health, writes it into its log manifest, and rejects a document or a log that names another.

Security

Three rules shape the security design. The first is that a request over its caller's quota comes back as an error saying so, rather than being cut down without comment to a size the quota allows. The second is that holding a token is not the same as being part of the cluster: machines inside the fleet authenticate each other with certificates issued by the cluster's own authority, and a token that lets an outside client ask a question grants it no access on the inside. The third is that the key used to sign score cutoffs on the fast channel is a different key from the one clients hold, so a leaked client token cannot be used to forge a cutoff and truncate somebody's search.

In practice that means TLS on every listener with mutual TLS between the coordinator and its nodes, plaintext served on loopback only unless an operator opts in on purpose, bearer principals carrying their own max_k, concurrency, and ingest-rate quotas, and signature tags on the cutoff and cancel datagrams. On the reference fleet, a plaintext dial and a certificate-less TLS dial both time out at the node, a TLS dial without the bearer returns an error naming the missing header, and the acceptance matrix passes over TLS.

Cluster operations

Every shard writes each change to a log before it applies it, which is an old and dependable idea: the log is the record of what happened, and the index files are a view of it that can be rebuilt. Growing or shrinking a cluster leans on that. A parent shard is copied at a known point in its log; the children are built from that copy in the background while the parent keeps serving queries and accepting new documents; each child then reads forward through the log from the copy point, applying only the writes that fall in its own range; and when the children have caught up, the coordinator switches to a new numbered version of the map that says which shard holds what. No data moves in the middle of a query.

Compaction, which merges many small index files into fewer large ones, runs the same way. On the reference fleet both Raspberry Pi shards were compacted from 102 sealed segments down to 64, one per log bucket, while continuing to answer queries throughout: 2 hours 37 minutes on one board and 3 hours 14 minutes on the other, of which the window where writes had to wait was 97 ms and 141 ms. The full acceptance suite passed 14 of 14 immediately afterwards. One thing to know about compaction is that it renumbers a shard's rows, which matters if you are holding ground truth keyed to row positions.

An optional control service owns leases, capacity reports, placement, topology history, and publication, with policy expressed as thresholds: lease duration, replication factor, the row count that triggers a split, the combined row count that triggers a merge, the segment count and tombstone fraction that trigger a compaction. Ranking and data movement stay in the coordinator and node paths.

Embedded and mobile

The embedded package runs the same public service contract over private local shards through an in-process coordinator. It is a local cluster, not a federated query: the coordinator reaches each shard through a direct call into the same node handler, framing a client-streaming call into the handler's body in memory, not over a socket. So there is no second ranking implementation to keep in step, and the crate links no HTTP/2 and no networking at all, which a dependency-tree check in its own tests keeps true. It supports create and open, protobuf-mapped ingest, vector ingest, delete and replace, the query and streaming-query routes, health, and flush, on Android, iOS, and desktop.

Related