High-level API: nind.nind_engine#
The modern, pure-Python entry point for indexing and searching a corpus. See Quickstart for a walk-through.
High-level nind frontend: index text files and search them with BM25.
This module is the modern entry point for the nind package: unlike
NindFile/NindPadFile and the Nind*index
classes (which only read the nind binary formats, as ergonomic Python
wrappers over the nind._native bindings where available - see the
package’s CLAUDE.md), NindIndexer here is the only Python code
that writes .nindlexiconindex/.nindtermindex/.nindlocalindex
files, and NindEngine provides a simple BM25 search API built on
top of the read-only classes.
Typical usage:
from nind.nind_engine import NindIndexer, NindEngine
NindIndexer("indices", prefix="corpus").index_files(["a.py", "b.py"])
NindIndexer("indices", prefix="docs").index_jsonl("corpus.jsonl") # pyserini's format
with NindIndexer("indices", prefix="corpus").update() as update:
update.add_file("c.py") # incremental: add, replace, remove
engine = NindEngine("indices")
for doc_id, score in engine.search("def foo"):
print(doc_id, score)
NindEngine#
- class NindEngine(index_dir, prefix=None, search_threads=1, map_lengths=False, directory_cache_mb=1024)[source]#
Bases:
objectBM25 search engine on top of a nind index produced by
NindIndexer.Opens the three index files (lexicon, term, local) for a corpus and exposes term/document-frequency lookups plus a ready-to-use
search(). Read-only: to build the index files this reads, useNindIndexer.Safe to use while the index is being rebuilt, and from several threads: each call answers from one complete build of the index, and switches to a newer build as soon as one is published - nothing to reopen or close. Calls from several threads run in parallel (the native reads are thread-safe and release the GIL); a call started before a new build is published finishes on the build it started with. The low-level objects returned by
lexicon,term_indexandlocal_indexare thread-safe for their lookups (donneIdentifiant,donneListeTermesCG,donneListeTermes), not for their diagnostic methods (dumpeFichier,afficheTerme…).Open the index in
index_dir.- Parameters:
index_dir – directory the index was written into by
NindIndexer(or containing the flat.nindlexiconindex/.nindtermindex/.nindlocalindexfiles of an index written by an older version).prefix – the index’s filename prefix; only needed when
index_dirholds several indexes.search_threads – threads each
search()uses by default (itsthreadsargument overrides it).1(the default) suits a server answering many queries at once: run them in parallel instead, every call releases the GIL. More lowers the latency of a single query on idle cores (MS MARCO, 8.8M documents, top 10, median: 1.5 ms with 1 thread, 0.9 ms with 4), for block-max searches only (atop_ksmall for the corpus).map_lengths – read the documents’ lengths where they are, in their mapped files (shared by every process searching the index: none of this process’s memory, 0.5 GB less for MS MARCO v2’s 138M passages), instead of a copy (searches 3-6% faster). For many search processes on one index.
directory_cache_mb – the most memory, in MiB, the query terms’ directories (their blocks’ descriptors) take once read, shared by the index’s segments; past it the terms used longest ago are dropped and read again when a query needs them. A query set of MS MARCO v2 (138M passages or 10.96M documents) takes ~0.75 GB of them: at 256 MB, searches of long documents took twice as long.
- Raises:
FileNotFoundError – if there is no index in
index_dir.ValueError – if there are several and no
prefixis given.
- property lexicon#
The
NindLexiconindexof the current build, if it is one index (elseValueError; seeNindIndexer.force_merge()).
- property term_index#
The
NindTermindexof the current build, if it is one index (elseValueError; seeNindIndexer.force_merge()).
- property local_index#
The
NindLocalindexof the current build, if it is one index (elseValueError; seeNindIndexer.force_merge());Nonefor an index built withlocal_index=False.
- property total_docs#
Number of documents in the index.
- property avg_doc_len#
Average document length (in tokens) across the corpus.
- tokenize(text)[source]#
Split source-code-like text into tokens, as the index’s documents were.
Uses the tokenizer options the index was built with (see
NindIndexer): queries must be tokenized like the corpus for search results to be meaningful.- Parameters:
text – the text to tokenize.
- Returns:
the list of tokens.
- get_term_id(term)[source]#
Look up a term’s internal lexicon identifier.
- Parameters:
term – the (single, non-compound) term.
- Returns:
its identifier, or
0if the term is not in the lexicon.
- get_tf(term, doc_id)[source]#
Get the raw term frequency of
termin a specific document.- Parameters:
term – the term to look up.
doc_id – the document’s external identifier.
- Returns:
the number of occurrences (summed across grammatical categories), or
0if the term is unknown or absent from that document.
- get_df(term)[source]#
Get the document frequency of
term(how many documents contain it).- Parameters:
term – the term to look up.
- Returns:
the number of documents containing the term, or
0if it is unknown.
- get_doc_len(doc_id)[source]#
Get the length (total token occurrences) of a document.
- Parameters:
doc_id – the document’s external identifier.
- Returns:
the document’s length in tokens.
- doc_name(doc_id)[source]#
The name of a document: its identifier in the corpus (e.g. the
idof its record forNindIndexer.index_jsonl()).- Parameters:
doc_id – the document’s identifier in the index.
- Returns:
its name, or
Noneif it has none.
- doc_id(name)[source]#
The identifier in the index of the document named
name(seedoc_name()).- Returns:
the document’s identifier, or
Noneif no document has that name.
- bm25_score(query, doc_id, k1=1.5, b=0.75)[source]#
Compute the Okapi BM25 relevance score of a document for a query.
- Parameters:
query – the query text (tokenized with
tokenize()).doc_id – the document’s external identifier.
k1 – BM25 term-frequency saturation parameter.
b – BM25 document-length normalization parameter.
- Returns:
the BM25 score (higher is more relevant);
0.0if no query term occurs in the document.
- search(query, top_k=10, doc_names=False, k1=1.5, b=0.75, threads=None)[source]#
Search the corpus with BM25 and return the top-scoring documents.
- Parameters:
query – the query text.
top_k – maximum number of results to return.
doc_names – return the documents’ names (see
doc_name();Nonefor a document without name) instead of their identifiers in the index - like pyserini’s hits’docid.k1 – BM25 term-frequency saturation parameter.
b – BM25 document-length normalization parameter.
threads – threads searching this one query at once (default: the engine’s
search_threads): the index’s segments in parallel, and for atop_ksmall for the corpus, more threads than segments within them, with the same results. For many queries at once, callingsearchfrom several threads scales better (each call releases the GIL).
- Returns:
a list of
(doc_id, score)tuples ((name, score)withdoc_names), sorted by descending score, of length at mosttop_k(only documents with a positive score are included).
Only the documents containing a query term are scored, from the terms’ postings lists (in native code): the cost depends on how many documents contain the query terms, not on the size of the corpus. Scores are those of
bm25_score()(to rounding: the native code computes the same formula, not the same operations); equal scores keep the order of the documents in the index.
NindIndexer#
- class NindIndexer(index_dir, prefix='corpus', tokenizer=None, local_index=None)[source]#
Bases:
objectBuilds a nind binary index (lexicon + term + local files) from text files.
The only Python writer for the nind index-family formats (see the module docstring): it builds them via
nind._native’sis_writer=Trueconstructors.Two ways of indexing:
index_files()(orindex_jsonl()) (re)builds the whole index, in parallel;update()(and its shortcutsadd_file(),add_files(),remove_documents()) changes an existing index - or creates one - incrementally.Texts are tokenized by the native tokenizer (
nind._native.Tokenizer), with the options given at construction, stored with the index so thatNindEnginetokenizes queries the same way. To tokenize differently, a subclass may override_default_tokenize()(a function of a text returning its list of tokens): it is then used instead, in pure Python (slower).Prepare an indexer that will write into
index_dir.Does not touch the filesystem until it indexes.
- Parameters:
index_dir – directory the index files will be written into (must already exist).
prefix – filename prefix shared by the index files (e.g.
"corpus"->corpus.nindlexiconindexetc.).tokenizer – tokenizer options, a dict overriding some of
TOKENIZER_DEFAULTS:split_camel_case,split_underscore,split_digits(each kind of split of identifiers is optional),lowercase,min_lengthandmax_length(in bytes; longer or shorter tokens are dropped). Default: the defaults for a new index, and the index’s own options for anupdate()(which refuses explicitly different ones).local_index – whether to write the local index (per document: its terms and their positions). The search engine doesn’t need it (it only needs document lengths, stored apart), and it makes up most of an index’s size and a good part of its build time; other readers of nind indexes (e.g. Amose) may need it. Default: with a local index for a new index, the index’s own choice for an
update()(which refuses an explicitly different one).
- index_files(file_paths, workers=None, shards=1, doc_names=None, merge=False, memory_gb=None)[source]#
Tokenize
file_pathsand write the resulting corpus as a nind index.Each file becomes one document, identified externally by its 0-based position in
file_paths. Replaces any existing index with the same prefix inindex_dir.Always safe: the new index is built apart and only published, at once, when complete. Until then,
NindEngineinstances keep answering from the previous index (and switch to the new one by themselves); if indexing fails, the previous index stays in place. Severalindex_fileson the same index wait for each other.Files are tokenized in parallel worker processes, then the lexicon is written, then the term and local index files (independent of each other once the lexicon is written) are written concurrently, each by its own single writer, in bulk mode.
- Parameters:
file_paths – list of paths to UTF-8 (or UTF-8-decodable, with errors ignored) text files to index.
workers – number of processes tokenizing the files (default:
os.cpu_count()).1, or a corpus of fewer thanPARALLEL_MIN_FILESfiles, tokenizes in this process instead, since starting worker processes would then cost more than it saves.shards – number of independent indexes (each over a contiguous part of
file_paths) built in parallel processes, at mostworkersat a time, which become the index’s segments.1builds the index directly. Seeindex_jsonl().merge – with
shards > 1, merge them into one index. Default: they stay segments (seeindex_jsonl()).doc_names – the documents’ names (their identifiers in the corpus, unique), in
file_pathsorder;NindEngine.doc_name()andsearch(..., doc_names=True)return them. Default: none.memory_gb – the memory the build may use, in GiB, as in
index_jsonl(): the tokens and postings estimated from a few runs of consecutive files spread overfile_paths, scaled by the files’ sizes (not counted:doc_namesand the list of paths, which the caller holds anyway).
- index_jsonl(path, id_field='id', text_field='contents', limit=None, workers=None, shards=1, merge=False, memory_gb=None)[source]#
Index a JSON Lines corpus (one document per line) as a nind index.
The format read by pyserini’s
JsonCollection: each non-blank line is a JSON object such as{"id": "d49309", "contents": "..."}. Then-th record becomes documentn(0-based), as then-th file does inindex_files(); the records’ own identifiers are returned, in document order, for the caller to map results back. Replaces any existing index, with the same safety guarantees asindex_files().The file is parsed and tokenized in parallel worker processes, each reading its own line-aligned part of it.
Large corpora (more than about 10M documents): give
memory_gb, orshards. A direct build holds the whole corpus in memory (about 8 bytes per token, 35 per document and 100 per distinct term: 2.9 GiB for 8M MS MARCO passages; about 5 per token when its postings are built in 4 passes, asmemory_gbmay do; seeFIX_MMAP_THRESHOLD) and is slower; shards bound the memory and build in parallel, and their segments give the same results.- Parameters:
path – path to the UTF-8
.jsonlfile.id_field – name of the field holding a record’s identifier.
text_field – name of the field holding the text to index.
limit – index only the first
limitrecords (default: all).workers – number of processes parsing and tokenizing the file (default:
os.cpu_count());1, or a file smaller thanJSONL_PARALLEL_MIN_BYTES, works in this process.shards – number of independent indexes, each over a line-aligned part of the file of about the same bytes, built in worker processes (at most
workersat a time), which become the index’s segments: the same search results as a direct build, searched in parallel withthreads, and no time spent merging (MS MARCO 8.8M: 25 s with 8 or 32 shards, 45 s direct; searches on one thread cost 16% more with 8, 69% with 32; with 8 threads, nothing more). This is the way to index a corpus too large to build at once: the peak is aboutworkerstimes a shard’s build (see above).merge – with
shards > 1, merge them into one index at the end (the merge’s memory: ~30 bytes per document, plus the longest term’s postings at 16 bytes each, not counted bymemory_gb; shards and merged index on disk together).memory_gb – the memory the build may use, in GiB: the fewest shards (at least
shards; one, a direct build, if it fits) whose builds,workersat a time beside this process, fit inMEMORY_GB_MARGINof it, estimated from the tokens, postings, documents and vocabulary of a few samples of the file (BUILD_BYTES_PER_*,HEAPS_EXPONENT); each build’s postings in up toMAX_BUILD_PASSESranges of terms, the fewest that fit (a scan of its tokens each, about 2 s for 16M passages). Fewer workers if each wouldn’t get twiceBUILD_BYTES_PER_PROCESS; fewer workers also mean fewer shards, i.e. faster searches on one thread.
- Returns:
the
id_fieldvalue of every indexed record: elementnis the identifier of documentn. A read-only sequence of them as strings, read from the index files at the first access (no list of every id in memory); a list of the values themselves for a direct build with ids that aren’t all strings (numbers).- Raises:
ValueError – if a line isn’t a JSON object with both fields.
- FIX_MMAP_THRESHOLD = True#
A direct build (
index_jsonl(),index_files()) fixes glibc’s mmap threshold at its default in this process, for good (a third less memory: see _fix_mmap_threshold); False leaves it dynamic (the planner’s model then underestimates a direct build, ~1.4 times). Worker processes always fix it.
- JSONL_PARALLEL_MIN_BYTES = 1048576#
Below this many bytes of JSON Lines,
index_jsonl()works in-process.
- JSONL_SHARD_PIECE_BYTES = 67108864#
A shard of
index_jsonl()parses at most this many bytes of the file at a time.
- BUILD_BYTES_PER_TOKEN = 4.0#
The memory of one build (direct, or a shard’s), as
index_jsonl(memory_gb=...)estimates it: bytes per token, per posting (a document’s distinct term: the postings built at once, the whole build’s divided by its passes, seeIndexBuilder.postings_budget), per document, per distinct term (HEAPS_EXPONENT: a build’s vocabulary grows as its tokens to this power, from the samples’ vocabulary) and per process. MS MARCO v2, shards of 1M to 9.5M passages built by a pool worker, 1 to 5 passes: 0 to 6% over (1M: 14%).
- BUILD_BYTES_PER_KEPT_POSTING = 5.0#
Without a local index, a build keeps its documents compact (
IndexBuilder.keep_tokens): per posting (each document’s distinct terms) and per posting of a frequency above 1, in place ofBUILD_BYTES_PER_TOKEN. Shards built by a pool worker, 1 and 4 passes: MS MARCO v2.1 documents (0.2M to 1.2M) 0.98 to 1.08 of the model, v2 passages (1M to 9.5M) 1.04 to 1.21 (1M passages, one pass: 1.43)
- BUILD_BYTES_PER_REPEATED = 4.0#
- BUILD_BYTES_PER_POSTING = 5#
- BUILD_BYTES_PER_DOCUMENT = 35#
- BUILD_BYTES_PER_TERM = 100#
- BUILD_BYTES_PER_PROCESS = 262144000#
- HEAPS_EXPONENT = 0.7#
- MAX_BUILD_PASSES = 4#
At most this many passes over a build’s tokens, each building the postings of a range of terms: a pass costs a scan of the tokens (2 s for 16M passages), and from about 4 the tokenizing phase is the peak.
- MEMORY_GB_MARGIN = 0.9#
equal shards started together reach their peaks together (138M, 8 workers, 16 GiB: 20 shards, 12.7 to 13.0 GiB from one run to another).
- Type:
index_jsonl(memory_gb=...)plans its builds in this part of the budget
- JSONL_SAMPLE_PIECES = 8#
index_jsonl(memory_gb=...)counts the tokens, documents and distinct terms of this many pieces of the file, of this many bytes each, spread over it (the whole file when smaller).
- JSONL_SAMPLE_BYTES = 2097152#
- force_merge()[source]#
Merge the index’s segments into one standard index, its deleted documents dropped.
Searches on one thread are at their fastest then, and
NindEngine.lexiconand the other direct accesses work. Costs a rewrite of the index; published at once, as a build.- Returns:
what was done (
nind._native.SegmentedUpdateStats).
- update()[source]#
Start an incremental update of the index (created if there is none yet).
An index written in format v1 (by a nind older than format v2) stays readable as a segment; merges rewrite it in format v2.
- Returns:
an
IndexUpdate, to use as a context manager; its changes are published together when thewithblock ends.
- add_files(file_paths, doc_ids=None, workers=None, doc_names=None)[source]#
Add (or replace) documents incrementally: one
update()for all offile_paths.- Parameters:
file_paths – paths of the text files to index, one document each.
doc_ids – their identifiers (an existing document with the same id is replaced); default: new ids following the highest one, or the document with the same name.
workers – processes tokenizing the files, as in
index_files().doc_names – their names (see
IndexUpdate.add_file()).
- Returns:
the documents’ identifiers, in
file_pathsorder.
- add_file(path, doc_id=None, name=None)[source]#
Add (or replace) one document incrementally; see
add_files().Each call publishes a new version of the index, which costs a copy of the index: to add many files, use
add_files()or oneupdate()for all of them.- Returns:
the document’s identifier.
- remove_documents(doc_ids)[source]#
Remove documents from the index, in one
update().- Returns:
for each id, whether there was such a document.
- PARALLEL_MIN_FILES = 64#
Below this many files,
index_files()tokenizes in-process.
- CHUNK_MAX_FILES = 4096#
At most this many files per tokenizing task.
IndexUpdate#
- class IndexUpdate(indexer)[source]#
Bases:
objectOne batch of incremental changes to an index, published at once when it ends.
Obtained from
NindIndexer.update()and used as a context manager:with indexer.update() as update: for path in paths: doc_id = update.add_file(path) update.remove(3)
Documents can be added (or replaced: adding an id the index already has replaces that document) and removed; the index may not exist yet. Nothing is visible until the
withblock ends without an exception: the changes are then published in one step,NindEngineinstances switching to them by themselves. If the block raises, the index is left as it was. An update holds the index’s writer lock from start to end: other updates andNindIndexer.index_files()on the same index wait.An update costs what it changes, whatever the size of the index: its documents are written into a new segment, the replaced and removed ones are masked in theirs, the published segments stay as they are (hard links); segments are merged now and then (see
MERGE_FACTOR), which makes each document be rewritten about log(N) times over its life. The results are those of an index built at once from the live documents.Documents are tokenized as the index’s were (its stored tokenizer options); an index built without local index stays without (replacing or removing documents then reads every term’s directory in their segments).
Not thread-safe: use an update from one thread at a time.
- MERGE_FACTOR = 10#
more segments cost searches more (MS MARCO 8.8M, one thread: +16% with 8, +69% with 32; with 8 threads, nothing).
0: never merge (seeNindIndexer.force_merge()).- Type:
Segments of a size tier merged into one (tiers grow by this factor)
- MAX_MERGED_DOCS = 5000000#
No merge into a segment of more live documents than this; at half of it a segment is full and stays as it is (rewriting it costs more than it gains: a large build’s segments are left alone).
0: no limit.
- MAX_MERGED_BYTES = 536870912#
a merge’s cost is its postings, and long documents make large segments of few documents (MS MARCO v2.1, 32 segments of 342k web pages: the first update of 1% merged 9.8M of them, in 1,689 s). 512 MiB: ~5M MS MARCO passages.
0: no limit.- Type:
Nor into a segment of more term index bytes than this (its live share)
- stats#
added_docs_nb,removed_docs_nb,segments_nb,merges_nb,merged_docs_nb),Nonebefore.- Type:
What the last commit did (
nind._native.SegmentedUpdateStats
- add_file(path, doc_id=None, name=None)[source]#
Index the text file
pathas one document.- Parameters:
path – path to a UTF-8 (or UTF-8-decodable, with errors ignored) text file.
doc_id – the document’s identifier; replaces the document with this id if there is one. Default: the document named
nameif there is one, else one more than the highest id ever given in the index or in this update (0for the first document).name – the document’s name (see
NindEngine.doc_name()), unique in the index: adding a document with the name of another one replaces it (unlessdoc_idis another document: thenValueError). Default: a replaced document keeps its name.
- Returns:
the document’s identifier.
- add_text(text, doc_id=None, name=None)[source]#
Index
textas one document; seeadd_file().
- add_texts(texts, doc_ids=None, names=None)[source]#
Index several texts, as
add_text()each, tokenized natively in one batch: nothing is done per token in Python (the way to add many documents from memory;add_jsonl()from a file).- Parameters:
doc_ids – the documents’ identifiers (
None, orNoneitems: asadd_text()’s default).names – the documents’ names (
None, orNoneitems: a replaced document keeps its name).
- Returns:
the documents’ identifiers, in order.
- add_tokens(tokens, doc_id=None, name=None)[source]#
Index an already tokenized document; see
add_file().A document without any token isn’t stored (an existing document with that id is removed). Tokens a lexicon can’t store (empty, or longer than 254 bytes in UTF-8) are dropped.
- add_jsonl(path, id_field='id', text_field='contents', workers=None)[source]#
Add the records of a JSON Lines corpus (the format of
NindIndexer.index_jsonl()), each named by itsid_field: a record with the name of a document replaces it (asadd_tokens()withname), the others get new identifiers.The way to add many documents: the file is parsed and tokenized in worker processes, and the documents go to the C++ API in batches, nothing done per token in Python.
- Parameters:
workers – processes parsing and tokenizing the file (default:
os.cpu_count());1, or a file smaller thanNindIndexer.JSONL_PARALLEL_MIN_BYTES, works in this process.- Returns:
the documents’ identifiers, in the file’s order.
- Raises:
ValueError – if a line isn’t a JSON object with both fields.