Milvus full text search with nind#
Milvus answers full text search with a BM25 function: a
VARCHAR field’s analyzer feeds a sparse vector field, which Milvus indexes and
searches. NindMilvusClient replaces that function with nind: it is a
pymilvus.MilvusClient in which nind does all the BM25 work, so code written for
Milvus full text search runs unchanged.
Only the client’s class changes. Compared with Milvus’s own BM25, nind gives the same
results, answers searches returning only ids 2 to 10 times faster, and keeps its
statistics exact after updates and deletions (see the results below).
An example module, not part of the wheel
NindMilvusClient lives in the repository’s
examples/milvus/
directory (nind_milvus.py, one file), not in the nind package: copy it into your
project. It needs nind 4.2.0 or later (for add_texts())
and pymilvus 3.0, and targets a Milvus 3.0 server (tested with 3.0.6). It is a proof
of concept: read its limits below before using it in production.
Set up#
pip install "nind>=4.2" "pymilvus[milvus-lite]==3.0.3"
A Milvus server: the example directory has a script that runs Milvus 3.0.6 standalone in Docker (embedded etcd, local storage, its data in a Docker volume):
examples/milvus/milvus-standalone.sh start # http://127.0.0.1:19530 (MILVUS_PORT to change it)
examples/milvus/milvus-standalone.sh stop # purge: also removes its data
Milvus Lite (a local file as uri, installed with pymilvus) works too, but its
built-in BM25 isn’t the server’s (see “Differences” below).
Use it#
The code of Milvus’s full text search documentation, with NindMilvusClient in place
of MilvusClient and a directory for nind’s indexes:
from pymilvus import AnnSearchRequest, DataType, Function, FunctionType, RRFRanker
from nind_milvus import NindMilvusClient
client = NindMilvusClient(uri="http://localhost:19530", nind_dir="nind-indexes")
schema = client.create_schema()
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("text", DataType.VARCHAR, max_length=65535, enable_analyzer=True)
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=768)
schema.add_function(Function(name="bm25", function_type=FunctionType.BM25,
input_field_names=["text"], output_field_names=["sparse"]))
index_params = client.prepare_index_params()
index_params.add_index(field_name="sparse", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25",
params={"bm25_k1": 1.2, "bm25_b": 0.75})
index_params.add_index(field_name="dense", index_type="AUTOINDEX", metric_type="COSINE")
client.create_collection("docs", schema=schema, index_params=index_params)
client.insert("docs", rows) # upsert and delete (ids or filter) alike
hits = client.search("docs", data=["binary search tree"], anns_field="sparse", limit=10,
filter='lang == "en"', output_fields=["text"])
for hit in hits[0]:
print(hit.id, hit.distance, hit["text"])
client.hybrid_search("docs", [AnnSearchRequest(data=["binary search tree"], anns_field="sparse",
param={}, limit=50),
AnnSearchRequest(data=[query_vector], anns_field="dense",
param={}, limit=50)],
ranker=RRFRanker(60), limit=10, output_fields=["text"])
Clients of one Milvus must share nind_dir (a shared filesystem when they run on
several machines). Many small writes are cheaper grouped: each write call is one nind
update, which costs about 0.25 s to publish, whatever its size:
with client.bm25_batch(): # one nind update for all of them
for batch in batches:
client.insert("docs", batch)
How it works#
create_collectiontakes the BM25 functions out of the schema. Each function’s output field stays, with the same name, as a nullable sparse vector nothing fills (Milvus loads no collection without a vector field), so code searchinganns_field="sparse"keeps working. nind indexes the function’s input field, one index per function undernind_dir/<database>/<collection>/, with thebm25_k1andbm25_bof the field’s BM25 index (Milvus’s defaults: 1.2 and 0.75).create_index,drop_collectionandrename_collectionfollow on nind’s side.Writes go to Milvus, then to nind, in one
IndexUpdateper call: the rows are tokenized natively in one batch (add_texts()), and the update is published atomically, searches switching to it by themselves. nind’s document ids are the primary keys (INT64, negative ones included), or nind document names for VARCHAR keys. Withauto_id, they are the ids Milvus returns.delete(filter=...)reads the keys the filter selects, then deletes them by id in both. A partial upsert without the text leaves it as it is.searchon a BM25 output field is answered by nind (block-max MaxScore, see Benchmarks). The hits are pymilvusHits (hit.id,hit.distance,hit["field"]). Afilter(orpartition_names) is applied by Milvus to nind’s ranking, aspk in [...] and (filter), read further down until enough rows pass.output_fieldsare read from Milvus, in one query.offsetand per-searchbm25_k1/bm25_b(insearch_params["params"]) are supported.hybrid_search: the BM25 requests go to nind, the others to Milvus, and the lists are fused client-side as Milvus’sRRFRankerandWeightedRankerdo (WeightedRankernormalizes BM25 and IP scores with arctan, COSINE with (1+x)/2).The analyzer. Milvus’s default analyzer (
standard) is reproduced by nind’s tokenizer: runs of letters and digits (err_4012giveserrand4012,don'tgivesdonandt,utf8stays whole), lowercased (Unicode, by the client, since nind’s tokenizer lowercases ASCII only). This was checked with Milvus’srun_analyzer. Thewhitespacetokenizer and thelowercasefilter are supported too. Other analyzers (stemming, stop words, languages) raiseNotImplementedError; passNindMilvusClient(tokenizer=...)with nind tokenizer options instead (see Quickstart).Repair. Writes are not transactional (Milvus first, then nind). If a write fails in between,
client.rebuild_bm25("docs")rebuilds the collection’s nind indexes from its rows. It also attaches nind to rows written by another client.
Results#
examples/milvus/compare.py loads a corpus into two collections with the same schema,
one with Milvus’s BM25 and one with nind’s, then searches each query in both (top 10,
one query at a time, from Python). These runs used Milvus 3.0.6 standalone on 48 cores,
with the corpora of bench/retrieval (lowercased, Lucene stop words removed, except
the full MS MARCO).
corpus |
documents |
nDCG@10, Milvus / nind |
same top 10 ranking |
|---|---|---|---|
nfcorpus |
3,633 |
0.3079 / 0.3079 |
99.7% |
scifact |
5,183 |
0.6645 / 0.6645 |
100% |
fiqa |
57,600 |
0.2380 / 0.2380 |
99.5% |
MS MARCO, first 1M passages |
1,000,000 |
(no judgments) |
99.1% |
MS MARCO (dev, 2,000 queries) |
8,841,823 |
0.2317 / 0.2317 |
99.8% (same set: 100%) |
The remaining differences are ties (equal scores, in another order) and float rounding. Latency, median / 95th percentile in ms:
fiqa |
MS MARCO 1M |
MS MARCO 8.8M |
|
|---|---|---|---|
Milvus BM25, ids |
7.19 / 9.01 |
7.43 / 9.77 |
11.08 / 20.65 |
nind, ids |
0.46 / 1.06 |
0.72 / 2.75 |
4.95 / 19.84 |
Milvus BM25, with the text |
7.60 / 9.49 |
7.94 / 11.72 |
11.70 / 20.95 |
nind, with the text |
8.60 / 10.99 |
8.81 / 17.13 |
11.94 / 28.55 |
Insertion until searchable took 7.5 s with Milvus’s BM25 and 7.7 s with nind on fiqa, 56.7 s and 57.6 s on MS MARCO 1M, and 572 s and 735 s on MS MARCO 8.8M (nind merging its segments as it goes, with one update per 500,000 rows).
These figures show two things:
Ids only: the BM25 search itself is 2 to 10 times faster in nind. On this server any Milvus request costs at least 5 to 7 ms, even a query of 10 rows by id.
With fields: returning fields costs nind one Milvus query, about one Milvus round trip, so a search with
output_fieldsis on par with Milvus’s own, or slightly slower.
The gain is for searches needing ids only (or fields kept elsewhere), and for hybrid pipelines that fuse ids first and read the fields once.
To reproduce them (queries.tsv, optional qrels.txt and corpus.jsonl as
bench/retrieval/prepare.py writes them):
python examples/milvus/compare.py DATA_DIR --uri http://127.0.0.1:19530 --queries 2000
python examples/milvus/demo.py # one scenario with both clients, results compared
Differences with Milvus’s BM25#
Statistics of live documents. After an upsert or a delete, Milvus’s BM25 still counts the replaced and deleted rows in the number of documents, the average length and the document frequencies (in our tests, even after a manual flush and compaction). nind counts the live rows only: its scores are those of an index rebuilt from them. Before any replacement or deletion, the scores are the same, to float precision.
Not supported with a BM25 field:
rankerandfunction_chainsinsearch, highlighters,group_by_field, search byids, rerankers other than RRF and weighted inhybrid_search, sparse vectors given as queries, and analyzers other than standard, whitespace and lowercase.text_matchis not BM25: Milvus does it, withenable_matchon the field.Consistency. A nind search sees every published nind update at once. The rows read for
output_fieldsandfilterfollow the collection’s consistency level, or the search’sconsistency_level.Milvus Lite’s built-in BM25 differs from the server’s: it returns negated scores and keeps
err_4012whole. nind follows the server’s. On Lite, the client fills the placeholder sparse field with a dummy vector, because Lite refuses a missing one.
Limits#
Not transactional: Milvus is written first. If a write fails before nind is updated, run
rebuild_bm25.One writer per collection at a time: nind serializes its updates with a lock file. Searches run in parallel.
Filters and fields cost a Milvus query: a filtered search, or one returning fields, makes one more Milvus query (
pk in [...]). Storing the fields in nind, or running nind inside Milvus’s query nodes, would avoid it.Databases: the client uses the
db_nameit was created with.using_databaseisn’t followed.