Low-level API: binary format readers

Read-only classes mirroring nind’s on-disk formats, layered as described in Architecture. Their hot-path lookups and shared structural diagnostics (analyseFichierPadFile/analyseFichierIndex) delegate to the compiled nind._native extension; only the remaining per-format diagnostic/introspection methods (dumpeFichier and similar) still parse the binary format directly in Python. They can open files written by either this package’s own NindIndexer or the C++ implementation.

nind.NindFile

Low-level binary codec for nind’s “Latecon” number/string encodings.

NindFile is the base of every other class in this package: it wraps a plain file handle and adds readers/writers for the primitive types used throughout nind’s binary formats (fixed-width big/little-endian integers, variable-length “ULat”/”SLat” integers, and length-prefixed UTF-8 strings). Every other reader (NindPadFile, NindIndex, …) builds on top of it.

This module is read/write-oriented but does no structural interpretation of the files it reads or writes - that is the job of nind.NindPadFile and the classes built on it. See the <fichier> grammar comment below for the primitive types it implements.

clefA(mot)

Compute the “A” hash key of a word (French: clefA = “key A”).

A simple rolling XOR/shift hash over the word’s UTF-8 bytes. Used where nind needs a second, differently-shifted hash from the same word (clefB is the other one) - see the C++ NindLexicon header comments for the rationale.

Parameters:

mot – the word to hash.

Returns:

the hash as an unsigned integer.

clefB(mot)

Compute the “B” hash key of a word (French: clefB = “key B”).

Same idea as clefA() with different shift constants. This is the hash NindLexiconindex actually uses to bucket words: a word’s entry lives at index clefB(mot) % nombreIndirection in the .nindlexiconindex file.

Parameters:

mot – the word to hash.

Returns:

the hash as an unsigned integer.

catNb2Str(cat)

Translate a grammatical-category code into its short label.

French: donne la catégorie grammaticale en clair = “give the grammatical category in clear [text]”. The numeric codes are the <catégorie> values stored in .nindtermindex/.nindlocalindex entries (Amose’s term-type model, see LAT2015.JYS.448).

Parameters:

cat – the numeric category code.

Returns:

the category’s short label (e.g. "ADJ", "NC", "V"), or '' if cat is out of range.

class NindFile(latFileName, enEjcriture=False, enModification=False)

Bases: object

Binary file wrapper implementing nind’s “Latecon” primitive codecs.

Wraps a single OS file handle opened in one of three modes (read-only, write-from-scratch, or read/modify-in-place) and exposes matching pairs of litXxx (French lit = “reads”) / ejcritXxx (French écrit = “writes”, spelled with the project’s “ej” transliteration of é) methods for each primitive type in the grammar above: fixed-width 1/3/4/5-byte integers (some little-endian/”petit-boutiste”, some big-endian/”gros-boutiste” - see each method), the ULat/SLat variable-length integer encodings, and length-prefixed or raw UTF-8 byte strings.

Every higher-level nind reader (NindPadFile and its subclasses) subclasses this and only adds structure on top - none of them re-implement byte-level (de)serialization.

Usable as a context manager (with NindFile(...) as f:), which guarantees close() is called.

Open latFileName in one of three modes.

Parameters:

• latFileName – path to the file to open.

• enEjcriture – if True, open for writing (French enÉcriture = “in writing [mode]”); the file is truncated and written from scratch unless enModification is also set. If False (the default), the file is opened read-only.

• enModification – if True (and enEjcriture is also True), open an existing file for in-place read/write modification (French enModification = “in modification [mode]”) instead of truncating it.

seek(offset, from_what)

Move the file position, exactly like io.IOBase.seek().

Parameters:

• offset – byte offset, interpreted relative to from_what.

• from_what – 0 = from the start, 1 = from the current position, 2 = from the end (same convention as the stdlib).

tell()

Return the current byte position in the file.

litNombre1()

Read and return an unsigned 1-byte integer (0..255).

litNombre2()

Read and return an unsigned 2-byte big-endian integer.

litNombre3()

Read and return an unsigned 3-byte little-endian integer.

litNombreS3()

Read and return a signed (two’s-complement) 3-byte little-endian integer.

litNombre4()

Read and return an unsigned 4-byte little-endian integer.

litNombreS4()

Read and return a signed (two’s-complement) 4-byte little-endian integer.

litNombre5()

Read and return an unsigned 5-byte big-endian integer.

Used for file offsets (e.g. <offsetDéfinition> in index files), which need more than 4 bytes of range.

litNombreULat()

Read and return an unsigned “ULat” variable-length integer.

This is nind’s own variable-length encoding (French Latecon, the project’s original code-name): the top bits of the first byte say how many extra continuation bytes follow (0 to 4), so small values take 1 byte and the encoding scales up to a full unsigned 32-bit value in 5 bytes. Used wherever a count or a delta is expected to usually be small (term/document frequencies, relative doc ids, …).

Raises:

Exception – if the encoding is malformed.

litNombreSLat()

Read and return a signed “SLat” variable-length integer.

The signed counterpart of litNombreULat(): same self-describing variable-length scheme, but reserving one bit per tier for the sign so it can encode negative deltas (e.g. relative term-id or position deltas that can go backwards).

Raises:

Exception – if the encoding is malformed.

litString()

Read a length-prefixed UTF-8 string (French litString, mixed EN/FR name).

Reads one length byte (0..255) followed by that many bytes, decoded as UTF-8 - i.e. the <MotUtf8> grammar rule.

Returns:

the decoded string.

litChaine(longueur)

Read longueur bytes and decode them as UTF-8 (French litChaine = “reads string”).

Like litString() but the caller supplies the length (already read separately), rather than a leading length byte.

Parameters:

longueur – number of bytes to read.

Returns:

the decoded string.

litOctets(longueur)

Read and return longueur raw bytes, undecoded (French litOctets = “reads bytes”).

ejcritNombre1(entier)

Write entier as an unsigned 1-byte integer.

Parameters:

entier – value in 0..255.

ejcritNombre3(entier)

Write entier as an unsigned 3-byte little-endian integer (see litNombre3()).

Parameters:

entier – value in 0..0xFFFFFF.

ejcritNombre4(entier)

Write entier as an unsigned 4-byte little-endian integer (see litNombre4()).

Parameters:

entier – value in 0..0xFFFFFFFF.

ejcritNombre5(entier)

Write entier as an unsigned 5-byte big-endian integer (see litNombre5()).

Parameters:

entier – value in 0..0xFFFFFFFFFF.

ejcritNombreULat(entier)

Write entier using the unsigned “ULat” variable-length encoding (see litNombreULat()).

Parameters:

entier – non-negative value; must fit in 32 bits.

Raises:

ValueError – if entier is too large to encode.

ejcritNombreSLat(entier)

Write entier using the signed “SLat” variable-length encoding (see litNombreSLat()).

Parameters:

entier – signed value; must fit in 32 bits.

Raises:

ValueError – if entier is too large to encode.

ejcritChaine(chaine)

Write chaine as raw UTF-8 bytes, with no length prefix (French ejcritChaine = “writes string”).

Callers that need a length prefix write it separately (see how NindIndexer pairs this with ejcritNombre1() of len(encoded) to build a <MotUtf8>).

Parameters:

chaine – the string to write.

ejcritZejros(taille)

Write taille zero bytes (French ejcritZéros = “writes zeros”).

Used to reserve/pad a region (e.g. an indirection table or the specifics+identification trailer) before patching it with real values once their final offsets are known.

Parameters:

taille – number of zero bytes to write.

close()

Close the underlying file handle.

nind.NindPadFile

The shared “pad file” envelope used by every nind index/lexicon file.

A “pad file” (the name reflects that entries are padded to a fixed size so they can be located by simple arithmetic) is nind’s common on-disk container: a small fixed header, one or more indexed blocks (fixed-size indirection tables mapping an integer identifier to an offset+length elsewhere in the file), the variable-length definitions those indirections point to (“en vrac”, i.e. loosely packed), and a trailer holding format-specific data plus a max-identifier/timestamp identification stamp. See the <fichier> grammar comment below for the exact layout.

NindPadFile implements this envelope read-only (mirroring NindBasics::NindPadFile in C++, which also has no writer - every concrete writer, in C++ or in NindIndexer here, hand-rolls the envelope itself). Concrete formats (NindIndex and its .nindlexiconindex/.nindtermindex/.nindlocalindex subclasses, NindRetrolexicon) subclass it and only add the meaning of the “en vrac” definitions and the specifics block.

class NindPadFile(padFileName, enEjcriture=False, enModification=False, tailleEntreje=0, tailleSpejcifiques=0)

Bases: NindFile

Read-only base class implementing the pad-file envelope over NindFile.

Subclasses get, for free, the machinery to locate an entry by identifier (donnePositionEntreje()), find the file’s specifics and identification trailer (donneSpejcifiques(), donneIdentificationFichier()), and walk/validate the whole file (analyseFichierPadFile()).

Open (or create) a pad file.

Parameters:

• padFileName – path to the file.

• enEjcriture – if True, create the file for writing and write its fixed header (tailleEntreje/tailleSpejcifiques) plus zeroed specifics+identification; if False (default), open read-only and read that same fixed header back.

• enModification – passed through to NindFile (in-place modification mode).

• tailleEntreje – size in bytes of one indirection-table entry (French tailleEntrée = “entry size”); only used when writing.

• tailleSpejcifiques – size in bytes of the format-specific trailer block (French tailleSpécifiques = “specifics size”); only used when writing.

vejrifieFichier()

Check the file’s structural integrity (French vérifieFichier = “checks file”).

Builds the indirection-blocks map (raising if the FLAG_INDEXEJ markers are missing/misplaced) and confirms the specifics block’s FLAG_SPEJCIFIQUE marker is where expected. Subclasses’ __init__ call this right after opening, so that a malformed file fails fast rather than on first lookup.

Raises:

Exception – if the file’s structure is invalid.

donneIdentificationFichier()

Return the file’s identification trailer (French donneIdentificationFichier = “gives file identification”).

Returns:

a (maxIdentifiant, identifieurUnique) tuple: the highest identifier ever assigned in this file, and a unique stamp (in practice a Unix timestamp) written when the file was finalized.

Raises:

Exception – if the trailer’s FLAG_IDENTIFICATION marker is missing.

donneSpejcifiques()

Locate the format-specific trailer block (French donneSpécifiques = “gives specifics”).

The “specifics” block holds whatever extra fixed-size data a concrete format needs beyond the generic envelope (e.g. NindLocalindex stores the document count there).

Returns:

a (offsetSpejcifiques, tailleSpejcifiques) tuple: the byte offset of the block and its size (as declared in the file’s fixed header).

Raises:

Exception – if the block’s FLAG_SPEJCIFIQUE marker is missing.

donnePositionEntreje(ident)

Return the file offset of the indirection-table entry for ident.

French donnePositionEntrée = “gives entry position”. Looks up ident across the (possibly chained) indexed blocks built by vejrifieFichier(); if not found, rebuilds the blocks map once (in case the file grew since it was opened) before giving up.

Parameters:

ident – the identifier to locate.

Returns:

the byte offset of that identifier’s indirection entry, or 0 if ident is out of range.

donneTailleFichier()

Return the total size of the file in bytes (French donneTailleFichier = “gives file size”).

donneMaxIdentifiant()

Return the highest identifier addressable by this file’s index blocks.

French donneMaxIdentifiant = “gives max identifier”. This is the capacity of the indirection system (sum of nombreIndex across all chained indexed blocks), not necessarily the highest identifier actually in use - compare with donneIdentificationFichier()’s maxIdentifiant, which is.

Raises:

Exception – if a block’s FLAG_INDEXEJ marker is missing.

donneCarteNonVides()

Return the file’s known “non-empty” (occupied) byte ranges.

French donneCarteNonVides = “gives non-empty map”. Lists the ranges occupied by the fixed header and every indexed block, as a starting point for callers that then add the ranges occupied by the definitions themselves, so chercheVides() can find what’s left over (“holes” - freed or never-written space).

Returns:

a (maxIdent, nonVidesList) tuple: the addressable identifier capacity (same as donneMaxIdentifiant()), and a list of (address, length) occupied ranges.

Raises:

Exception – if a block’s FLAG_INDEXEJ marker is missing.

changeIdentificationFichier(maxIdentifiant, identifieurUnique)

Overwrite the file’s identification trailer in place.

French changeIdentificationFichier = “changes file identification”. The file must have been opened in a writable mode (enEjcriture/enModification); this seeks to the trailer and rewrites its two fields.

Parameters:

• maxIdentifiant – new highest-identifier value to record.

• identifieurUnique – new unique stamp to record.

Raises:

Exception – if the trailer’s FLAG_IDENTIFICATION marker is missing.

analyseFichierPadFile(trace)

Walk and validate the whole pad-file envelope, optionally printing a report.

French analyseFichierPadFile = “analyzes pad file”. Delegates the binary traversal to nind._native’s analyse_pad_file() (see NindPadFile::analysePadFile in the C++), then reports the same size breakdowns as before (fixed header / indexed blocks / “en vrac” data / specifics / identification). Used both as a diagnostic (Nind_dumpDocument.py and friends) and as the first step of every subclass’s own analyseFichierXxx method.

Parameters:

trace – if True, print a human-readable report to stdout.

Returns:

True if the envelope is structurally valid, False otherwise (errors are reported via trace rather than raised).

Note:

unlike the previous hand-rolled implementation, a structural error stops the report at that point rather than continuing to print whatever the remaining sections can still compute.

chercheVides(nonVidesList)

Derive the “holes” (unused gaps) between a file’s occupied byte ranges.

French chercheVides = “looks for empties”. Sorts nonVidesList by address and walks it, reporting any gap between consecutive occupied ranges as a “vide” (hole) - typically caused by entries that were deleted or replaced by a bigger definition without compacting the file. Used by every analyseFichierXxx/statistics method to report on file fragmentation.

Parameters:

nonVidesList – list of (address, length) occupied ranges (as produced by e.g. NindPadFile.donneCarteNonVides()); sorted in place.

Returns:

a (nbreVides, tailleVides, typesVides, typesNonVides) tuple: the number and total size of holes found, and two {length: count} histograms (of hole sizes and of occupied-range sizes, respectively).

Raises:

Exception – if two occupied ranges overlap.

calculeRejpartition(nombres)

Compute basic descriptive statistics over a list of numbers.

French calculeRépartition = “computes distribution”. Small helper shared by the various analyseFichierXxx methods to summarize things like definition sizes or term frequencies.

Parameters:

nombres – list of numbers (int or float).

Returns:

a (count, min, max, sum, mean, stddev) tuple; all zeros if nombres is empty.

nind.NindIndex

Generic “definitions indexed by integer identifier” pad file.

NindIndex specializes NindPadFile for the common case where the indirection table maps a simple integer identifier directly to a variable-length “definition” record: each indirection entry is just an (offset, length) pair (see the <indirection> grammar rule below). This is the shared shape behind all three concrete index formats: NindLexiconindex (.nindlexiconindex, keyed by hash bucket), NindTermindex (.nindtermindex, keyed by term id) and NindLocalindex (.nindlocalindex, keyed by internal document id) - each of those only adds the meaning of the bytes inside a definition.

class NindIndex(indexFileName)

Bases: NindPadFile

Read-only pad file whose indirection entries are plain (offset, length) pairs.

Adds donneAdresseDejfinition() on top of NindPadFile, resolving an identifier straight to where its variable-length definition lives; format-specific subclasses use it and then decode the definition’s bytes themselves.

Open indexFileName read-only and verify its structure.

Parameters:

indexFileName – path to the .nind*index file.

Raises:

Exception – if the file’s pad-file envelope is invalid.

donneAdresseDejfinition(identifiant)

Return the offset and length of identifiant’s definition.

French donneAdresseDéfinition = “gives definition address”.

Parameters:

identifiant – the integer identifier to look up.

Returns:

an (offsetDejfinition, longueurDejfinition) tuple, or (0, 0) if identifiant is out of range.

analyseFichierIndex(trace)

Validate the file (via analyseFichierPadFile()) and report indirection-usage statistics.

French analyseFichierIndex = “analyzes index file”. Delegates the indirection walk to nind._native’s analyse_index() (see NindIndex::analyseIndex in the C++), which checks that every identifier resolves to a valid indirection entry and reports the size distribution of definitions plus the holes between them.

Parameters:

trace – if True, print a human-readable report to stdout.

Returns:

True if the file is structurally valid, False otherwise.

Note:

unlike the previous hand-rolled implementation, a structural error stops the report at that point rather than continuing to print whatever the remaining sections can still compute.

nind.NindRetrolexicon

Reverse lexicon: word identifier -> clear-text word(s).

A .nindretrolexicon file is the mirror image of NindLexiconindex (which maps a word to its identifier): given an identifier, NindRetrolexicon returns the word it stands for, including reconstructing compound words from their component identifiers. It is a plain NindPadFile (not a NindIndex): entries here are either a UTF-8 word directly, or a pointer pair to two other entries (see the grammar below).

Not written by NindIndexer (NindEngine only supports simple, non-compound words and has no use for it) - this class exists to read files produced by the C++ side or by legacy nind tooling.

class NindRetrolexicon(retrolexiconFileName, lexicon_identification=None)

Bases: NindPadFile

Read-only reverse lexicon: resolves a word identifier back to text.

Each entry is either a “simple word” (a UTF-8 string stored directly in the file’s “en vrac” area) or a “compound word” (a pair of identifiers identifiantA/identifiantS pointing at the word’s two parts, themselves resolved recursively by donneMot()).

donneMot() delegates to the nind._native bindings (opened with a real lexicon_identification when given, or a “no cross-check” sentinel otherwise - a real lexicon isn’t required just to read this file’s own words) - dumpeFichier/analyseFichierRetrolexicon themselves call donneMot() and keep working standalone either way.

Open retrolexiconFileName read-only and verify its structure.

Parameters:

• retrolexiconFileName – path to the .nindretrolexicon file.

• lexicon_identification – the nind._native.Identification of the NindLexiconindex this retrolexicon was built against (see its donneIdentification). Optional: enables the nind._native-backed fast path in donneMot(); omit it to use the pure-Python fallback.

Raises:

Exception – if the file does not exist, or its pad-file envelope is invalid.

createFile()

No-op: this class is read-only and never creates a file.

Kept as a stub because sibling writer classes (in C++, and conceptually here) expose a createFile; there is no Python writer for this format (see the module docstring).

donneMot(ident)

Resolve a word identifier to its component simple word(s).

French donneMot = “gives word”. Follows the identifiantA / identifiantS chain for as long as the entry is a compound word, collecting each simple word it bottoms out on, in order.

Parameters:

ident – the word identifier to resolve.

Returns:

a list of simple-word strings (one element for a simple word, several for a compound word, in left-to-right order), or [] if ident is unknown.

Raises:

Exception – if the file is inconsistent (a dangling identifier, a non-terminal entry where a terminal one was expected, or a self-referential loop).

analyseFichierRetrolexicon(trace)

Validate the file and report simple/compound word statistics.

French analyseFichierRetrolexicon = “analyzes retrolexicon file”. Validates the pad-file envelope, then walks every identifier and tallies how many are simple vs. compound words, plus a byte-level breakdown (UTF-8 word bytes vs. holes) via chercheVides().

Parameters:

trace – if True, print a human-readable report to stdout.

Raises:

Exception – if the file is structurally invalid (also reported via trace before being re-raised).

dumpeFichier(outFile)

Dump every resolvable word to a text file, one id  word line each.

French dumpeFichier = “dumps file”. Compound words are written with their components joined by _. Used by the Nind_checkRejtroAndLexicon.py-style diagnostics to produce a human-readable lexicon.

Parameters:

outFile – a writable text file object (UTF-8).

Returns:

a (nbLignes, nbErreurs, rejpartition) tuple: number of lines written, number of those that hit an error, and a list of (wordLengthInComponents, count) pairs describing the distribution of compound-word lengths.

nind.NindLexiconindex

Lexicon-as-index: word (text) -> identifier, hash-bucketed.

A .nindlexiconindex file is a NindIndex whose identifiers are hash buckets (clefB(mot) % nombreIndirection, see clefB()) rather than direct word ids: each bucket’s definition packs every word that hashed into it, alongside its identifier and, for compound words, the identifiers of its components. This is the format NindIndexer writes as .nindlexiconindex and NindEngine reads to turn a query term into the identifier used to look it up in NindTermindex.

class NindLexiconindex(lexiconindexFileName)

Bases: NindIndex

Read-only word-to-identifier lexicon, hash-bucketed by clefB().

donneIdentifiant() and donneIdentification() delegate to the nind._native bindings; the rest (analyseFichierLexiconindex, dumpeFichier, debogueIndex, donneCollisions, donneMax, donneClef) are diagnostics used by Nind_* command-line tools to inspect hash-bucket distribution and collisions, and remain hand-rolled pure-Python parsing since nind._native doesn’t expose that introspection.

Open lexiconindexFileName read-only.

Parameters:

lexiconindexFileName – path to the .nindlexiconindex file.

Raises:

Exception – if the file’s pad-file envelope is invalid.

donneIdentifiant(motsSimples)

Look up the identifier of a word, given as a list of simple words.

French donneIdentifiant = “gives identifier”. A single-element list looks up a simple word; a multi-element list looks up the compound word formed by those simple words in order.

Parameters:

motsSimples – list of one or more simple-word strings, e.g. ["compulsive"] or ["épistémologie", "compulsive"].

Returns:

the word’s identifier, or 0 if it is not in the lexicon.

donneIdentification()

Return this lexicon’s nind._native.Identification.

Needed to open the corresponding .nindtermindex/ .nindlocalindex/.nindretrolexicon files, which cross-check against it.

analyseFichierLexiconindex(trace)

Validate the file and report simple/compound word statistics.

French analyseFichierLexiconindex = “analyzes lexicon-index file”. Extends analyseFichierIndex() with a breakdown of simple vs. compound words per bucket and the distribution of compound-word component counts.

Parameters:

trace – if True, print a human-readable report to stdout.

Returns:

False if the underlying index is invalid; otherwise prints the report when trace is set (no explicit return value on the success path).

dumpeFichier(outFile)

Dump every bucket’s entries to a text file, for inspection.

French dumpeFichier = “dumps file”. One line per word: its identifier, the word itself, and (for compound words) its component identifier pairs.

Parameters:

outFile – a writable text file object (UTF-8).

Returns:

a (nbLignes, nbErreurs) tuple: number of lines written, and number of those that hit a decoding error.

debogueIndex(index)

Print a step-by-step trace of decoding one bucket’s definition.

French debogueIndex = “debugs index”. Diagnostic helper for the Nind_checkRejtroAndLexicon.py CLI: prints every field as it is read, to help pinpoint where a corrupted bucket goes wrong.

Parameters:

index – the bucket (hash) index to inspect.

donneCollisions(index)

List every word hash-bucketed onto index (a “hash collision” report).

French donneCollisions = “gives collisions”.

Parameters:

index – the bucket (hash) index to inspect.

Returns:

a list of (motSimple, identifiantS, nbreComposes) tuples, one per word sharing that bucket.

donneMax(taille)

Find the words with the most components and the buckets with the most collisions.

French donneMax = “gives max[ima]”. Scans the whole file to find the top taille compound words by component count, and the top taille buckets by number of colliding words.

Parameters:

taille – how many top entries to keep for each ranking.

Returns:

a (composejs, collisions) tuple: composejs is a list of (nbreComposejs, identifiantS, index) tuples sorted by descending component count; collisions is a list of (nbreCollisions, index) tuples sorted by descending collision count.

donneClef(mot)

Return the bucket index a given simple word hashes to.

French donneClef = “gives key”. Debugging helper: same computation donneIdentifiant() performs internally, exposed so a caller can e.g. inspect a specific bucket with donneCollisions() or debogueIndex().

Parameters:

mot – the simple word to hash.

Returns:

its bucket index (clefB(mot) % nombreIndirection).

nind.NindTermindex

The inverted file itself: term identifier -> (category, frequency, postings).

A .nindtermindex file is a NindIndex keyed directly by term identifier (as assigned by NindLexiconindex). Each term’s definition holds one or more “CG” (catégorie grammaticale, grammatical-category) groups, each with the term’s total frequency in that category and its posting list - the (relative, delta-encoded) document ids and per-document frequencies where it occurs. This is what NindEngine reads to compute document/term frequencies for BM25 scoring.

class NindTermindex(termindexFileName, lexicon_identification=None)

Bases: NindIndex

Read-only inverted file: term identifier -> posting lists, grouped by grammatical category.

donneListeTermesCG() delegates to the nind._native bindings and requires lexicon_identification; the rest (analyseFichierTermindex, dumpeFichier, afficheTerme) are diagnostics used by the Nind_* command-line tools, work without a lexicon, and remain hand-rolled pure-Python parsing since nind._native doesn’t expose that introspection.

Open termindexFileName read-only.

Parameters:

• termindexFileName – path to the .nindtermindex file.

• lexicon_identification – the nind._native.Identification of the NindLexiconindex this term index was built against (see its donneIdentification). Required only for donneListeTermesCG(); the diagnostic methods work without it.

Raises:

Exception – if the file’s pad-file envelope is invalid.

donneListeTermesCG(ident)

Return the inverted-file entry for a term, grouped by grammatical category.

French donneListeTermesCG = “gives term list [by] grammatical category” (CG = catégorie grammaticale). Amose’s richer term-type model (LAT2015.JYS.448) allows the same term id to carry several categories (e.g. a word used as both noun and verb), each with its own frequency and posting list.

Parameters:

ident – the term identifier (as produced by NindLexiconindex).

Returns:

a list of nind._native.TermCG (.cg, .frequency, .documents: list of Document with .ident/ .frequency), one per grammatical category the term was seen in. Empty list if ident is unknown.

Raises:

Exception – if opened without lexicon_identification.

analyseFichierTermindex(trace)

Validate the file and report corpus-wide term/document statistics.

French analyseFichierTermindex = “analyzes term-index file”. Extends analyseFichierIndex() with totals (document count, term-document occurrences, hapax count) and a frequency distribution, plus a consistency check that each CG group’s posting-list frequencies sum to its declared term frequency.

Parameters:

trace – if True, print a human-readable report to stdout.

Returns:

False if the underlying index is invalid; otherwise prints the report when trace is set (no explicit return value on the success path).

Raises:

Exception – if a term’s posting-list frequencies are inconsistent with its declared total.

dumpeFichier(outFile)

Dump every term’s full posting lists to a text file, for inspection.

French dumpeFichier = “dumps file”.

Parameters:

outFile – a writable text file object (UTF-8).

Returns:

the number of terms written.

Raises:

Exception – if a term’s posting-list frequencies are inconsistent with its declared total.

afficheTerme(identifiant)

Format one term’s full posting lists as a human-readable string.

French afficheTerme = “displays term”. Same content as one dumpeFichier() line, computed for a single term.

Parameters:

identifiant – the term identifier to format.

Returns:

a formatted multi-line string, or "<id> : inconnu" if identifiant is unknown.

Raises:

Exception – if the term’s posting-list frequencies are inconsistent with its declared total.

nind.NindLocalindex

The per-document local index: document identifier -> term occurrences and positions.

A .nindlocalindex file is a NindIndex keyed by internal document id (a dense 1..nombreDocuments sequence), plus an external <-> internal id translation table built at open time from each document’s stored identifiantExterne. Each document’s definition lists every term that occurs in it (delta-encoded relative term ids) together with the positions (“localisations”) it occurs at - the data NindEngine uses for term-frequency and document-length calculations.

class NindLocalindex(localindexFileName, lexicon_identification=None)

Bases: NindIndex

Read-only per-document local index: term occurrences and positions, keyed by document id.

Documents are addressed by their external id everywhere in the public API (donneListeTermes(), afficheDocument()); the class transparently translates to/from the internal id used inside the file, via a table built once at open time. donneListeTermes() delegates to the nind._native bindings and requires lexicon_identification; the diagnostic methods (analyseFichierLocalindex, dumpeFichier, afficheDocument) work without it, remaining hand-rolled pure-Python parsing since nind._native doesn’t expose that introspection.

Open localindexFileName read-only and build the external<->internal id table.

Parameters:

• localindexFileName – path to the .nindlocalindex file.

• lexicon_identification – the nind._native.Identification of the NindLexiconindex this local index was built against (see its donneIdentification). Required only for donneListeTermes(); the diagnostic methods work without it.

Raises:

Exception – if the file’s pad-file envelope is invalid, or its specifics block has the wrong size.

donneListeTermes(noDocExterne)

Return every term occurrence and its positions for a given document.

French donneListeTermes = “gives term list”.

Parameters:

noDocExterne – the document’s external identifier.

Returns:

a list of nind._native.Term (.term, .cg, .localisation: list of Localisation with .position/.length), one per term occurring in the document. Empty list if noDocExterne is unknown.

Raises:

Exception – if opened without lexicon_identification.

donneMaxIdentifiants()

Return the internal and external identifiers of the last-indexed document.

French donneMaxIdentifiants = “gives max identifiers”.

Returns:

an (internalId, externalId) tuple, or (0, 0) if that document has since been erased.

analyseFichierLocalindex(trace)

Validate the file and report corpus-wide document/occurrence statistics.

French analyseFichierLocalindex = “analyzes local-index file”. Extends analyseFichierIndex() with totals (document count, term-document and term-position occurrence counts) and a per-document occurrence-count distribution.

Parameters:

trace – if True, print a human-readable report to stdout.

Returns:

False if the underlying index is invalid; otherwise prints the report when trace is set (no explicit return value on the success path).

dumpeFichier(outFile)

Dump every document’s full term/position data to a text file, for inspection.

French dumpeFichier = “dumps file”.

Parameters:

outFile – a writable text file object (UTF-8).

Returns:

the number of documents written.

donneidentifiantsExternes()

Return every (externalId, internalId) pair known to this file.

French donneIdentifiantsExternes = “gives external identifiers”. Used by NindEngine to enumerate the whole corpus (e.g. to compute the total document count and average document length).

Returns:

a list of (noDocExterne, noDocInterne) tuples.

afficheDocument(noDocExterne)

Format one document’s full term/position data as a human-readable string.

French afficheDocument = “displays document”. Same content as one dumpeFichier() line, computed for a single document.

Parameters:

noDocExterne – the document’s external identifier.

Returns:

a formatted string, or "<id> : inconnu" if noDocExterne is unknown.