Quickstart#
The fastest way to use nind from Python is the high-level API in
nind.nind_engine: NindIndexer builds a
binary index from a list of files, and NindEngine
opens that index and answers BM25-ranked search queries.
Index a corpus#
from nind.nind_engine import NindIndexer
indexer = NindIndexer(index_dir="indices", prefix="corpus")
indexer.index_files([
"src/py/nind/NindFile.py",
"src/py/nind/NindPadFile.py",
"src/py/nind/nind_engine.py",
])
This writes the index into a new generation directory inside indices/
(corpus.gen-<n>-<hex>/: corpus.nindlexiconindex, corpus.nindtermindex,
corpus.nindlocalindex, plus corpus.nindmeta - the tokenizer options -,
corpus.ninddoclengths and, when documents have names, corpus.ninddocnames),
published atomically through corpus.CURRENT.
index_dir must already exist; each input file becomes one document,
identified externally by its position in the list (0, 1, 2, …).
A corpus in pyserini’s JSON Lines format (one {"id": ..., "contents": ...}
object per line) is indexed directly, record n becoming document n:
record_ids = indexer.index_jsonl("corpus.jsonl") # record_ids[n]: the "id" of document n
The records’ ids are stored with the index as the documents’ names
(index_files takes names as doc_names=[...]), so search results can be
reported with them, like pyserini’s hits’ docid - see
“Document names” below.
Tokenizer and index options#
Texts are tokenized by a native tokenizer for source-code-like text: words
are runs of letters and digits, split at camelCase (fooBar -> foo Bar,
HTTPServer -> HTTP Server), underscore (foo_bar -> foo bar) and
letter/digit (utf8 -> utf 8) boundaries. Each split is optional, and
tokens can be lowercased:
indexer = NindIndexer("indices", prefix="corpus",
tokenizer={"lowercase": True, "split_digits": False})
The options are stored with the index: NindEngine tokenizes queries the
same way, and updates keep them. Tokens longer than 254 bytes (the longest
word a nind lexicon stores) are dropped.
local_index=False leaves out the local index (each document’s terms and
their positions). NindEngine doesn’t need it - document lengths are stored
apart - and it makes up most of an index’s size and a good part of its build
time; other readers of nind indexes (Amose…) may need it:
indexer = NindIndexer("indices", prefix="corpus", local_index=False)
Index incrementally#
index_files rebuilds the whole index. To add, replace or remove documents
in an existing index instead - or to build one file by file - group the
changes in an update; they are published together when the with block
ends (nothing changes if it raises):
with indexer.update() as update:
for path in new_files:
doc_id = update.add_file(path) # new id: one more than the highest
update.add_file("changed.py", doc_id=1) # replaces document 1
update.remove(2)
update.add_text("...", name="d49309") # replaces the document named d49309, if any
A document’s name is unique in the index: adding a document under a name
that is already used replaces that document (giving another document’s id
with it raises ValueError); a replaced document keeps its name unless a new
one is given. update.doc_id(name) finds a document by name, changes of the
update included.
Many documents in memory go faster as one batch, tokenized natively with
nothing done per token in Python (add_jsonl(path) does the same from a file):
with indexer.update() as update:
update.add_texts(texts, doc_ids=ids) # or names=[...]; as add_text for each
This is how Milvus full text search with nind keeps a nind index in step with a Milvus collection.
add_file(path), add_files(paths) and remove_documents(ids) on the
indexer are one-update shortcuts. An update costs what it changes, whatever
the size of the index (about 0.25 s to publish, with its fsyncs, plus the
new documents): it writes a new segment and masks the replaced and removed
documents in theirs. Segments are merged now and then; the results are always
those of one index of the live documents. indexer.force_merge() makes it one
index, which a search on a single thread answers fastest, and which
engine.get_term_id, engine.lexicon, engine.term_index and
engine.local_index need (they raise ValueError on several segments or
deleted documents).
A large corpus (more than about 10M documents): indexer.index_jsonl(path, memory_gb=16) builds the fewest segments that fit in 16 GiB (a direct build if
the corpus fits), in worker processes, in parallel; shards=64 asks for 64
instead, with memory bounded by workers of them at a time. A direct build
holds the whole corpus (about 8 bytes per token, 35 per document and 100 per
distinct term; memory_gb may build its postings in up to 4 passes over the
tokens, about 5 bytes per token) and is slower; segments give the same results, and merge=True
(or a later indexer.force_merge()) makes one index of them.
Search it#
from nind.nind_engine import NindEngine
engine = NindEngine("indices")
for doc_id, score in engine.search("ejcrit nombre", top_k=5):
print(f"{doc_id}: {score:.3f}")
NindEngine auto-discovers the index by scanning index_dir for a
*.nindlexiconindex file, so it only needs the directory - not the prefix.
Search speed and threads#
A search reads only the postings it needs: for a top k small for the corpus, the blocks of postings that can’t reach the top k are skipped (block-max MaxScore). On MS MARCO (8.8 million passages) one query’s top 10 takes about 1.5 ms on one thread (median).
Many queries at once (a server): call search from several threads on one NindEngine. It
is thread-safe and releases Python’s GIL, so the queries run in parallel on the cores.
One query faster (idle cores, latency matters): split each query across threads, for the whole engine or per call:
engine = NindEngine("indices", search_threads=4) # every search uses 4 threads
engine.search("binary search tree", top_k=10) # MS MARCO: about 0.9 ms instead of 1.5
engine.search("binary search tree", threads=1) # this one on a single thread
The results are the same. The default is 1: splitting a query costs more work in total (4
threads make it about 1.5–2× faster, not 4×), which a busy server would rather spend on other
queries. Splitting applies to block-max searches (a top_k small for the corpus, the usual
case); see the Benchmarks.
Document names#
When the index stores document names (index_jsonl, index_files(..., doc_names=...), or name= in updates), the engine maps them to and from
document ids, and search can return them instead of ids:
engine.search("binary search tree", doc_names=True) # [("d49309", 12.7), ...]
engine.doc_name(0) # "d49309"; None if the index has no names
engine.doc_id("d49309") # 0; None if there is no such document
Inspecting a document or term directly#
Beyond search, NindEngine exposes the building blocks it uses
internally, which are handy for debugging relevance or inspecting a
specific document:
engine.get_term_id("nombre") # -> internal lexicon identifier, or 0 (one index only)
engine.get_df("nombre") # -> how many documents contain it
engine.get_tf("nombre", doc_id=1) # -> occurrences in that one document
engine.get_doc_len(doc_id=1) # -> document length in tokens
Reading files written by the C++ implementation#
Every class NindEngine builds on
(NindLexiconindex,
NindTermindex,
NindLocalindex) can also open index files
produced by the C++ stack directly - they only need the binary files on
disk, not anything written by NindIndexer. NindTermindex and
NindLocalindex do need the lexicon’s identification stamp to look
anything up (a cross-check the underlying format itself requires), so open
the lexicon first:
from nind.NindLexiconindex import NindLexiconindex
from nind.NindTermindex import NindTermindex
lexicon = NindLexiconindex("/path/to/corpus.nindlexiconindex")
term_index = NindTermindex("/path/to/corpus.nindtermindex", lexicon.donneIdentification())
for term_cg in term_index.donneListeTermesCG(term_id):
print(term_cg.cg, term_cg.frequency, term_cg.documents)
donneListeTermesCG (and the equivalent lookup methods on the other
reader classes) return the nind._native objects directly - here, a list
of TermCG, each with .cg/.frequency/.documents (a list of
Document, each with .ident/.frequency) - rather than plain tuples.
See Architecture for how these lower-level classes relate to each other, and API reference for the full reference.