High-level API: nind.nind_engine#

The modern, pure-Python entry point for indexing and searching a corpus. See Quickstart for a walk-through.

High-level nind frontend: index text files and search them with BM25.

This module is the modern entry point for the nind package: unlike NindFile/NindPadFile and the Nind*index classes (which only read the nind binary formats, as ergonomic Python wrappers over the nind._native bindings where available - see the package’s CLAUDE.md), NindIndexer here is the only Python code that writes .nindlexiconindex/.nindtermindex/.nindlocalindex files, and NindEngine provides a simple BM25 search API built on top of the read-only classes.

Typical usage:

from nind.nind_engine import NindIndexer, NindEngine

NindIndexer("indices", prefix="corpus").index_files(["a.py", "b.py"])
NindIndexer("indices", prefix="docs").index_jsonl("corpus.jsonl")    # pyserini's format
with NindIndexer("indices", prefix="corpus").update() as update:
    update.add_file("c.py")             # incremental: add, replace, remove
engine = NindEngine("indices")
for doc_id, score in engine.search("def foo"):
    print(doc_id, score)

NindEngine#

class NindEngine(index_dir, prefix=None, search_threads=1, map_lengths=False, directory_cache_mb=1024)[source]#

Bases: object

BM25 search engine on top of a nind index produced by NindIndexer.

Opens the three index files (lexicon, term, local) for a corpus and exposes term/document-frequency lookups plus a ready-to-use search(). Read-only: to build the index files this reads, use NindIndexer.

Safe to use while the index is being rebuilt, and from several threads: each call answers from one complete build of the index, and switches to a newer build as soon as one is published - nothing to reopen or close. Calls from several threads run in parallel (the native reads are thread-safe and release the GIL); a call started before a new build is published finishes on the build it started with. The low-level objects returned by lexicon, term_index and local_index are thread-safe for their lookups (donneIdentifiant, donneListeTermesCG, donneListeTermes), not for their diagnostic methods (dumpeFichier, afficheTerme…).

Open the index in index_dir.

Parameters:
  • index_dir – directory the index was written into by NindIndexer (or containing the flat .nindlexiconindex/.nindtermindex/.nindlocalindex files of an index written by an older version).

  • prefix – the index’s filename prefix; only needed when index_dir holds several indexes.

  • search_threads – threads each search() uses by default (its threads argument overrides it). 1 (the default) suits a server answering many queries at once: run them in parallel instead, every call releases the GIL. More lowers the latency of a single query on idle cores (MS MARCO, 8.8M documents, top 10, median: 1.5 ms with 1 thread, 0.9 ms with 4), for block-max searches only (a top_k small for the corpus).

  • map_lengths – read the documents’ lengths where they are, in their mapped files (shared by every process searching the index: none of this process’s memory, 0.5 GB less for MS MARCO v2’s 138M passages), instead of a copy (searches 3-6% faster). For many search processes on one index.

  • directory_cache_mb – the most memory, in MiB, the query terms’ directories (their blocks’ descriptors) take once read, shared by the index’s segments; past it the terms used longest ago are dropped and read again when a query needs them. A query set of MS MARCO v2 (138M passages or 10.96M documents) takes ~0.75 GB of them: at 256 MB, searches of long documents took twice as long.

Raises:
property lexicon#

The NindLexiconindex of the current build, if it is one index (else ValueError; see NindIndexer.force_merge()).

property term_index#

The NindTermindex of the current build, if it is one index (else ValueError; see NindIndexer.force_merge()).

property local_index#

The NindLocalindex of the current build, if it is one index (else ValueError; see NindIndexer.force_merge()); None for an index built with local_index=False.

property total_docs#

Number of documents in the index.

property avg_doc_len#

Average document length (in tokens) across the corpus.

tokenize(text)[source]#

Split source-code-like text into tokens, as the index’s documents were.

Uses the tokenizer options the index was built with (see NindIndexer): queries must be tokenized like the corpus for search results to be meaningful.

Parameters:

text – the text to tokenize.

Returns:

the list of tokens.

get_term_id(term)[source]#

Look up a term’s internal lexicon identifier.

Parameters:

term – the (single, non-compound) term.

Returns:

its identifier, or 0 if the term is not in the lexicon.

get_tf(term, doc_id)[source]#

Get the raw term frequency of term in a specific document.

Parameters:
  • term – the term to look up.

  • doc_id – the document’s external identifier.

Returns:

the number of occurrences (summed across grammatical categories), or 0 if the term is unknown or absent from that document.

get_df(term)[source]#

Get the document frequency of term (how many documents contain it).

Parameters:

term – the term to look up.

Returns:

the number of documents containing the term, or 0 if it is unknown.

get_doc_len(doc_id)[source]#

Get the length (total token occurrences) of a document.

Parameters:

doc_id – the document’s external identifier.

Returns:

the document’s length in tokens.

doc_name(doc_id)[source]#

The name of a document: its identifier in the corpus (e.g. the id of its record for NindIndexer.index_jsonl()).

Parameters:

doc_id – the document’s identifier in the index.

Returns:

its name, or None if it has none.

doc_id(name)[source]#

The identifier in the index of the document named name (see doc_name()).

Returns:

the document’s identifier, or None if no document has that name.

bm25_score(query, doc_id, k1=1.5, b=0.75)[source]#

Compute the Okapi BM25 relevance score of a document for a query.

Parameters:
  • query – the query text (tokenized with tokenize()).

  • doc_id – the document’s external identifier.

  • k1 – BM25 term-frequency saturation parameter.

  • b – BM25 document-length normalization parameter.

Returns:

the BM25 score (higher is more relevant); 0.0 if no query term occurs in the document.

search(query, top_k=10, doc_names=False, k1=1.5, b=0.75, threads=None)[source]#

Search the corpus with BM25 and return the top-scoring documents.

Parameters:
  • query – the query text.

  • top_k – maximum number of results to return.

  • doc_names – return the documents’ names (see doc_name(); None for a document without name) instead of their identifiers in the index - like pyserini’s hits’ docid.

  • k1 – BM25 term-frequency saturation parameter.

  • b – BM25 document-length normalization parameter.

  • threads – threads searching this one query at once (default: the engine’s search_threads): the index’s segments in parallel, and for a top_k small for the corpus, more threads than segments within them, with the same results. For many queries at once, calling search from several threads scales better (each call releases the GIL).

Returns:

a list of (doc_id, score) tuples ((name, score) with doc_names), sorted by descending score, of length at most top_k (only documents with a positive score are included).

Only the documents containing a query term are scored, from the terms’ postings lists (in native code): the cost depends on how many documents contain the query terms, not on the size of the corpus. Scores are those of bm25_score() (to rounding: the native code computes the same formula, not the same operations); equal scores keep the order of the documents in the index.

NindIndexer#

class NindIndexer(index_dir, prefix='corpus', tokenizer=None, local_index=None)[source]#

Bases: object

Builds a nind binary index (lexicon + term + local files) from text files.

The only Python writer for the nind index-family formats (see the module docstring): it builds them via nind._native’s is_writer=True constructors.

Two ways of indexing: index_files() (or index_jsonl()) (re)builds the whole index, in parallel; update() (and its shortcuts add_file(), add_files(), remove_documents()) changes an existing index - or creates one - incrementally.

Texts are tokenized by the native tokenizer (nind._native.Tokenizer), with the options given at construction, stored with the index so that NindEngine tokenizes queries the same way. To tokenize differently, a subclass may override _default_tokenize() (a function of a text returning its list of tokens): it is then used instead, in pure Python (slower).

Prepare an indexer that will write into index_dir.

Does not touch the filesystem until it indexes.

Parameters:
  • index_dir – directory the index files will be written into (must already exist).

  • prefix – filename prefix shared by the index files (e.g. "corpus" -> corpus.nindlexiconindex etc.).

  • tokenizer – tokenizer options, a dict overriding some of TOKENIZER_DEFAULTS: split_camel_case, split_underscore, split_digits (each kind of split of identifiers is optional), lowercase, min_length and max_length (in bytes; longer or shorter tokens are dropped). Default: the defaults for a new index, and the index’s own options for an update() (which refuses explicitly different ones).

  • local_index – whether to write the local index (per document: its terms and their positions). The search engine doesn’t need it (it only needs document lengths, stored apart), and it makes up most of an index’s size and a good part of its build time; other readers of nind indexes (e.g. Amose) may need it. Default: with a local index for a new index, the index’s own choice for an update() (which refuses an explicitly different one).

index_files(file_paths, workers=None, shards=1, doc_names=None, merge=False, memory_gb=None)[source]#

Tokenize file_paths and write the resulting corpus as a nind index.

Each file becomes one document, identified externally by its 0-based position in file_paths. Replaces any existing index with the same prefix in index_dir.

Always safe: the new index is built apart and only published, at once, when complete. Until then, NindEngine instances keep answering from the previous index (and switch to the new one by themselves); if indexing fails, the previous index stays in place. Several index_files on the same index wait for each other.

Files are tokenized in parallel worker processes, then the lexicon is written, then the term and local index files (independent of each other once the lexicon is written) are written concurrently, each by its own single writer, in bulk mode.

Parameters:
  • file_paths – list of paths to UTF-8 (or UTF-8-decodable, with errors ignored) text files to index.

  • workers – number of processes tokenizing the files (default: os.cpu_count()). 1, or a corpus of fewer than PARALLEL_MIN_FILES files, tokenizes in this process instead, since starting worker processes would then cost more than it saves.

  • shards – number of independent indexes (each over a contiguous part of file_paths) built in parallel processes, at most workers at a time, which become the index’s segments. 1 builds the index directly. See index_jsonl().

  • merge – with shards > 1, merge them into one index. Default: they stay segments (see index_jsonl()).

  • doc_names – the documents’ names (their identifiers in the corpus, unique), in file_paths order; NindEngine.doc_name() and search(..., doc_names=True) return them. Default: none.

  • memory_gb – the memory the build may use, in GiB, as in index_jsonl(): the tokens and postings estimated from a few runs of consecutive files spread over file_paths, scaled by the files’ sizes (not counted: doc_names and the list of paths, which the caller holds anyway).

index_jsonl(path, id_field='id', text_field='contents', limit=None, workers=None, shards=1, merge=False, memory_gb=None)[source]#

Index a JSON Lines corpus (one document per line) as a nind index.

The format read by pyserini’s JsonCollection: each non-blank line is a JSON object such as {"id": "d49309", "contents": "..."}. The n-th record becomes document n (0-based), as the n-th file does in index_files(); the records’ own identifiers are returned, in document order, for the caller to map results back. Replaces any existing index, with the same safety guarantees as index_files().

The file is parsed and tokenized in parallel worker processes, each reading its own line-aligned part of it.

Large corpora (more than about 10M documents): give memory_gb, or shards. A direct build holds the whole corpus in memory (about 8 bytes per token, 35 per document and 100 per distinct term: 2.9 GiB for 8M MS MARCO passages; about 5 per token when its postings are built in 4 passes, as memory_gb may do; see FIX_MMAP_THRESHOLD) and is slower; shards bound the memory and build in parallel, and their segments give the same results.

Parameters:
  • path – path to the UTF-8 .jsonl file.

  • id_field – name of the field holding a record’s identifier.

  • text_field – name of the field holding the text to index.

  • limit – index only the first limit records (default: all).

  • workers – number of processes parsing and tokenizing the file (default: os.cpu_count()); 1, or a file smaller than JSONL_PARALLEL_MIN_BYTES, works in this process.

  • shards – number of independent indexes, each over a line-aligned part of the file of about the same bytes, built in worker processes (at most workers at a time), which become the index’s segments: the same search results as a direct build, searched in parallel with threads, and no time spent merging (MS MARCO 8.8M: 25 s with 8 or 32 shards, 45 s direct; searches on one thread cost 16% more with 8, 69% with 32; with 8 threads, nothing more). This is the way to index a corpus too large to build at once: the peak is about workers times a shard’s build (see above).

  • merge – with shards > 1, merge them into one index at the end (the merge’s memory: ~30 bytes per document, plus the longest term’s postings at 16 bytes each, not counted by memory_gb; shards and merged index on disk together).

  • memory_gb – the memory the build may use, in GiB: the fewest shards (at least shards; one, a direct build, if it fits) whose builds, workers at a time beside this process, fit in MEMORY_GB_MARGIN of it, estimated from the tokens, postings, documents and vocabulary of a few samples of the file (BUILD_BYTES_PER_*, HEAPS_EXPONENT); each build’s postings in up to MAX_BUILD_PASSES ranges of terms, the fewest that fit (a scan of its tokens each, about 2 s for 16M passages). Fewer workers if each wouldn’t get twice BUILD_BYTES_PER_PROCESS; fewer workers also mean fewer shards, i.e. faster searches on one thread.

Returns:

the id_field value of every indexed record: element n is the identifier of document n. A read-only sequence of them as strings, read from the index files at the first access (no list of every id in memory); a list of the values themselves for a direct build with ids that aren’t all strings (numbers).

Raises:

ValueError – if a line isn’t a JSON object with both fields.

FIX_MMAP_THRESHOLD = True#

A direct build (index_jsonl(), index_files()) fixes glibc’s mmap threshold at its default in this process, for good (a third less memory: see _fix_mmap_threshold); False leaves it dynamic (the planner’s model then underestimates a direct build, ~1.4 times). Worker processes always fix it.

JSONL_PARALLEL_MIN_BYTES = 1048576#

Below this many bytes of JSON Lines, index_jsonl() works in-process.

JSONL_SHARD_PIECE_BYTES = 67108864#

A shard of index_jsonl() parses at most this many bytes of the file at a time.

BUILD_BYTES_PER_TOKEN = 4.0#

The memory of one build (direct, or a shard’s), as index_jsonl(memory_gb=...) estimates it: bytes per token, per posting (a document’s distinct term: the postings built at once, the whole build’s divided by its passes, see IndexBuilder.postings_budget), per document, per distinct term (HEAPS_EXPONENT: a build’s vocabulary grows as its tokens to this power, from the samples’ vocabulary) and per process. MS MARCO v2, shards of 1M to 9.5M passages built by a pool worker, 1 to 5 passes: 0 to 6% over (1M: 14%).

BUILD_BYTES_PER_KEPT_POSTING = 5.0#

Without a local index, a build keeps its documents compact (IndexBuilder.keep_tokens): per posting (each document’s distinct terms) and per posting of a frequency above 1, in place of BUILD_BYTES_PER_TOKEN. Shards built by a pool worker, 1 and 4 passes: MS MARCO v2.1 documents (0.2M to 1.2M) 0.98 to 1.08 of the model, v2 passages (1M to 9.5M) 1.04 to 1.21 (1M passages, one pass: 1.43)

BUILD_BYTES_PER_REPEATED = 4.0#
BUILD_BYTES_PER_POSTING = 5#
BUILD_BYTES_PER_DOCUMENT = 35#
BUILD_BYTES_PER_TERM = 100#
BUILD_BYTES_PER_PROCESS = 262144000#
HEAPS_EXPONENT = 0.7#
MAX_BUILD_PASSES = 4#

At most this many passes over a build’s tokens, each building the postings of a range of terms: a pass costs a scan of the tokens (2 s for 16M passages), and from about 4 the tokenizing phase is the peak.

MEMORY_GB_MARGIN = 0.9#

equal shards started together reach their peaks together (138M, 8 workers, 16 GiB: 20 shards, 12.7 to 13.0 GiB from one run to another).

Type:

index_jsonl(memory_gb=...) plans its builds in this part of the budget

JSONL_SAMPLE_PIECES = 8#

index_jsonl(memory_gb=...) counts the tokens, documents and distinct terms of this many pieces of the file, of this many bytes each, spread over it (the whole file when smaller).

JSONL_SAMPLE_BYTES = 2097152#
force_merge()[source]#

Merge the index’s segments into one standard index, its deleted documents dropped.

Searches on one thread are at their fastest then, and NindEngine.lexicon and the other direct accesses work. Costs a rewrite of the index; published at once, as a build.

Returns:

what was done (nind._native.SegmentedUpdateStats).

update()[source]#

Start an incremental update of the index (created if there is none yet).

An index written in format v1 (by a nind older than format v2) stays readable as a segment; merges rewrite it in format v2.

Returns:

an IndexUpdate, to use as a context manager; its changes are published together when the with block ends.

add_files(file_paths, doc_ids=None, workers=None, doc_names=None)[source]#

Add (or replace) documents incrementally: one update() for all of file_paths.

Parameters:
  • file_paths – paths of the text files to index, one document each.

  • doc_ids – their identifiers (an existing document with the same id is replaced); default: new ids following the highest one, or the document with the same name.

  • workers – processes tokenizing the files, as in index_files().

  • doc_names – their names (see IndexUpdate.add_file()).

Returns:

the documents’ identifiers, in file_paths order.

add_file(path, doc_id=None, name=None)[source]#

Add (or replace) one document incrementally; see add_files().

Each call publishes a new version of the index, which costs a copy of the index: to add many files, use add_files() or one update() for all of them.

Returns:

the document’s identifier.

remove_documents(doc_ids)[source]#

Remove documents from the index, in one update().

Returns:

for each id, whether there was such a document.

PARALLEL_MIN_FILES = 64#

Below this many files, index_files() tokenizes in-process.

CHUNK_MAX_FILES = 4096#

At most this many files per tokenizing task.

IndexUpdate#

class IndexUpdate(indexer)[source]#

Bases: object

One batch of incremental changes to an index, published at once when it ends.

Obtained from NindIndexer.update() and used as a context manager:

with indexer.update() as update:
    for path in paths:
        doc_id = update.add_file(path)
    update.remove(3)

Documents can be added (or replaced: adding an id the index already has replaces that document) and removed; the index may not exist yet. Nothing is visible until the with block ends without an exception: the changes are then published in one step, NindEngine instances switching to them by themselves. If the block raises, the index is left as it was. An update holds the index’s writer lock from start to end: other updates and NindIndexer.index_files() on the same index wait.

An update costs what it changes, whatever the size of the index: its documents are written into a new segment, the replaced and removed ones are masked in theirs, the published segments stay as they are (hard links); segments are merged now and then (see MERGE_FACTOR), which makes each document be rewritten about log(N) times over its life. The results are those of an index built at once from the live documents.

Documents are tokenized as the index’s were (its stored tokenizer options); an index built without local index stays without (replacing or removing documents then reads every term’s directory in their segments).

Not thread-safe: use an update from one thread at a time.

MERGE_FACTOR = 10#

more segments cost searches more (MS MARCO 8.8M, one thread: +16% with 8, +69% with 32; with 8 threads, nothing). 0: never merge (see NindIndexer.force_merge()).

Type:

Segments of a size tier merged into one (tiers grow by this factor)

MAX_MERGED_DOCS = 5000000#

No merge into a segment of more live documents than this; at half of it a segment is full and stays as it is (rewriting it costs more than it gains: a large build’s segments are left alone). 0: no limit.

MAX_MERGED_BYTES = 536870912#

a merge’s cost is its postings, and long documents make large segments of few documents (MS MARCO v2.1, 32 segments of 342k web pages: the first update of 1% merged 9.8M of them, in 1,689 s). 512 MiB: ~5M MS MARCO passages. 0: no limit.

Type:

Nor into a segment of more term index bytes than this (its live share)

stats#

added_docs_nb, removed_docs_nb, segments_nb, merges_nb, merged_docs_nb), None before.

Type:

What the last commit did (nind._native.SegmentedUpdateStats

add_file(path, doc_id=None, name=None)[source]#

Index the text file path as one document.

Parameters:
  • path – path to a UTF-8 (or UTF-8-decodable, with errors ignored) text file.

  • doc_id – the document’s identifier; replaces the document with this id if there is one. Default: the document named name if there is one, else one more than the highest id ever given in the index or in this update (0 for the first document).

  • name – the document’s name (see NindEngine.doc_name()), unique in the index: adding a document with the name of another one replaces it (unless doc_id is another document: then ValueError). Default: a replaced document keeps its name.

Returns:

the document’s identifier.

add_text(text, doc_id=None, name=None)[source]#

Index text as one document; see add_file().

add_texts(texts, doc_ids=None, names=None)[source]#

Index several texts, as add_text() each, tokenized natively in one batch: nothing is done per token in Python (the way to add many documents from memory; add_jsonl() from a file).

Parameters:
  • doc_ids – the documents’ identifiers (None, or None items: as add_text()’s default).

  • names – the documents’ names (None, or None items: a replaced document keeps its name).

Returns:

the documents’ identifiers, in order.

add_tokens(tokens, doc_id=None, name=None)[source]#

Index an already tokenized document; see add_file().

A document without any token isn’t stored (an existing document with that id is removed). Tokens a lexicon can’t store (empty, or longer than 254 bytes in UTF-8) are dropped.

add_jsonl(path, id_field='id', text_field='contents', workers=None)[source]#

Add the records of a JSON Lines corpus (the format of NindIndexer.index_jsonl()), each named by its id_field: a record with the name of a document replaces it (as add_tokens() with name), the others get new identifiers.

The way to add many documents: the file is parsed and tokenized in worker processes, and the documents go to the C++ API in batches, nothing done per token in Python.

Parameters:

workers – processes parsing and tokenizing the file (default: os.cpu_count()); 1, or a file smaller than NindIndexer.JSONL_PARALLEL_MIN_BYTES, works in this process.

Returns:

the documents’ identifiers, in the file’s order.

Raises:

ValueError – if a line isn’t a JSON object with both fields.

doc_id(name)[source]#

The identifier of the document named name (None if there is none), changes of this update included.

remove(doc_id)[source]#

Remove the document doc_id from the index.

Returns:

True if there was such a document (in the index or added earlier in this update), False otherwise.