Handle multithread processing of token sequence. More...

#include <deeplima/token_sequence_analyzer.h>

Inheritance diagram for deeplima::ITokenSequenceAnalyzer:

Public Types
typedef std::function< void(std::shared_ptr< StringIndex > stridx, const token_buffer_t<> &tokens, const std::vector< StringIndex::idx_t > &lemmata, std::shared_ptr< StdMatrix< uint8_t > > classes, size_t begin, size_t end) >	output_callback_t

Public Member Functions
virtual void	register_handler (const output_callback_t fn)=0

virtual const std::vector< std::vector< std::string > > &	get_classes () const =0

virtual const std::vector< std::string > &	get_class_names () const =0

virtual std::shared_ptr< StringIndex >	get_stridx () const =0

virtual void	finalize ()=0

virtual void	operator() (const std::vector< segmentation::token_pos > &tokens, uint32_t len)=0

Detailed Description

Handle multithread processing of token sequence.

Uses several buffers. There is no dependency between buffers because we compute a little bit more than strictly necessary to serve as the context. After computing, not all the buffer is used as output.

Loads all necessary models

Precompute should be separated from inference during conversion of torch model to ours. But currently, the torch model is loaded during initialization

Definition at line 53 of file token_sequence_analyzer.h.

Member Typedef Documentation

◆ output_callback_t

typedef std::function< void (std::shared_ptr< StringIndex > stridx, const token_buffer_t<>& tokens, const std::vector<StringIndex::idx_t>& lemmata, std::shared_ptr< StdMatrix<uint8_t> > classes, size_t begin, size_t end) > deeplima::ITokenSequenceAnalyzer::output_callback_t

Definition at line 61 of file token_sequence_analyzer.h.