High-level API: nind.nind_engine#

The modern, pure-Python entry point for indexing and searching a corpus. See Quickstart for a walk-through.

High-level nind frontend: index text files and search them with BM25.

This module is the modern entry point for the nind package: unlike NindFile/NindPadFile and the Nind*index classes (which only read the nind binary formats, as ergonomic Python wrappers over the nind._native bindings where available - see the package’s CLAUDE.md), NindIndexer here is the only Python code that writes .nindlexiconindex/.nindtermindex/.nindlocalindex files, and NindEngine provides a simple BM25 search API built on top of the read-only classes.

Typical usage:

from nind.nind_engine import NindIndexer, NindEngine

NindIndexer("indices", prefix="corpus").index_files(["a.py", "b.py"])
engine = NindEngine("indices")
for doc_id, score in engine.search("def foo"):
    print(doc_id, score)

NindEngine#

class NindEngine(index_dir)[source]#

Bases: object

BM25 search engine on top of a nind index produced by NindIndexer.

Opens the three index files (lexicon, term, local) for a corpus and exposes term/document-frequency lookups plus a ready-to-use search(). Read-only: to build the index files this reads, use NindIndexer.

Open the index files found in index_dir.

Parameters:

index_dir – directory containing exactly one corpus’s .nindlexiconindex, .nindtermindex and .nindlocalindex files (same filename prefix).

Raises:

FileNotFoundError – if no .nindlexiconindex file is found in index_dir.

tokenize(text)[source]#

Split source-code-like text into tokens, splitting camelCase/snake_case identifiers into subwords.

Used both here (query tokenization for search()) and by NindIndexer (corpus tokenization at indexing time) - the two must stay in sync for search results to be meaningful.

Parameters:

text – the text to tokenize.

Returns:

a list of lowercase-preserving token strings.

get_term_id(term)[source]#

Look up a term’s internal lexicon identifier.

Parameters:

term – the (single, non-compound) term.

Returns:

its identifier, or 0 if the term is not in the lexicon.

get_tf(term, doc_id)[source]#

Get the raw term frequency of term in a specific document.

Parameters:

  • term – the term to look up.

  • doc_id – the document’s external identifier.

Returns:

the number of occurrences (summed across grammatical categories), or 0 if the term is unknown or absent from that document.

get_df(term)[source]#

Get the document frequency of term (how many documents contain it).

Parameters:

term – the term to look up.

Returns:

the number of documents containing the term, or 0 if it is unknown.

get_doc_len(doc_id)[source]#

Get the length (total token occurrences) of a document.

Parameters:

doc_id – the document’s external identifier.

Returns:

the document’s length in tokens.

bm25_score(query, doc_id, k1=1.5, b=0.75)[source]#

Compute the Okapi BM25 relevance score of a document for a query.

Parameters:

  • query – the query text (tokenized with tokenize()).

  • doc_id – the document’s external identifier.

  • k1 – BM25 term-frequency saturation parameter.

  • b – BM25 document-length normalization parameter.

Returns:

the BM25 score (higher is more relevant); 0.0 if no query term occurs in the document.

search(query, top_k=10)[source]#

Search the corpus with BM25 and return the top-scoring documents.

Parameters:

  • query – the query text.

  • top_k – maximum number of results to return.

Returns:

a list of (doc_id, score) tuples, sorted by descending score, of length at most top_k (only documents with a positive score are included).

NindIndexer#

class NindIndexer(index_dir, prefix='corpus')[source]#

Bases: object

Builds a nind binary index (lexicon + term + local files) from a list of text files.

The only Python writer for the nind index-family formats (see the module docstring): it builds them via nind._native’s is_writer=True constructors.

Prepare an indexer that will write into index_dir.

Does not touch the filesystem until index_files() is called.

Parameters:

  • index_dir – directory the index files will be written into (must already exist).

  • prefix – filename prefix shared by the three index files (e.g. "corpus" -> corpus.nindlexiconindex etc.).

index_files(file_paths)[source]#

Tokenize file_paths and write the resulting corpus as a nind index.

Each file becomes one document, identified externally by its 0-based position in file_paths. Overwrites any existing index files with the same prefix in index_dir.

Parameters:

file_paths – list of paths to UTF-8 (or UTF-8-decodable, with errors ignored) text files to index.