Text

class oceanai.modules.lab.text.TextMessages(lang: str = 'ru', color_simple: str = '#666', color_info: str = '#1776D2', color_err: str = '#FF0000', color_true: str = '#008001', bold_text: bool = True, text_runtime: str = '', num_to_df_display: int = 30)[source]

Bases: Download

Class for messages

Parameters:
class oceanai.modules.lab.text.Text(lang: str = 'ru', color_simple: str = '#666', color_info: str = '#1776D2', color_err: str = '#FF0000', color_true: str = '#008001', bold_text: bool = True, text_runtime: str = '', num_to_df_display: int = 30)[source]

Bases: TextMessages

Class for text processing

Parameters:
__load_bert_model(url: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the BERT model

Note

private method

Parameters:

  • url (str) – Full path to the file with the BERT model

  • force_reload (bool) – Forcing a file download from the BERT model from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the BERT model is downloaded, otherwise False

Return type:

bool

__load_model_weights(url: str, force_reload: bool = True, info_text: str = '', out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the weights of the neural network model

Note

private method

Parameters:

  • url (str) – Full path to the file with weights of the neural network model

  • force_reload (bool) – Forced download of a file with weights of a neural network model from the network

  • info_text (str) – Text for informational message

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the weights of the neural network model are downloaded, otherwise False

Return type:

bool

__load_text_features(url: str, force_reload: bool = True, info_text: str = '', out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the dictionary with hand-crafted features

Note

private method

Parameters:

  • url (str) – Full path to the file with hand-crafted features

  • force_reload (bool) – Forced download of a file with hand-crafted features from the network

  • info_text (str) – Text for informational message

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the hand-crafted feature dictionary is downloaded, otherwise False

Return type:

bool

__load_text_model_b5(show_summary: bool = False, out: bool = True) → Module | None

Formation of a neural network model architecture for obtaining the personality traits scores

Note

private method

Parameters:

  • show_summary (bool) – Displaying the formed neural network architecture of the model

  • out (bool) – Display

Returns:

None если неверные типы или значения аргументов, в обратном случае нейросетевая модель nn.Module для получения оценок персональных качеств

Return type:

Optional[nn.Module]

__process_audio_and_extract_features(path: str, win: int, lang: str, show_text: bool, last: bool, out: bool, url: str = None) → Tuple[ndarray, ndarray]
Parameters:

  • path (str)

  • win (int)

  • lang (str)

  • show_text (bool)

  • last (bool)

  • out (bool)

  • url (str)

Return type:

Tuple[ndarray, ndarray]

__translate_and_extract_features(text: str, lang: str, show_text: bool = False, last: bool = False, out: bool = True) → Tuple[ndarray, ndarray]

Extracting features from text

Note

private method

Parameters:

  • text (str) – Text

  • lang (str) – Language

  • show_text (bool) – Text display

  • last (bool) – Replacing the last message

  • out (bool) – Display

Returns:

Tuple with two np.ndarray: 1. np.ndarray with hand-crafted features 2. np.ndarray with deep features

Return type:

Tuple[np.ndarray, np.ndarray]

_get_text_features(path: str, asr: bool = False, lang: str = 'ru', show_text: bool = False, last: bool = False, out: bool = True, runtime: bool = True, run: bool = True) → Tuple[ndarray, ndarray][source]

Extracting features from text (without clearing the Jupyter cell’s message output history)

Note

protected method

Parameters:

  • path (str) – Path to video file or text

  • asr (bool) – Automatic speech recognition

  • lang (str) – Language

  • show_text (bool) – Text display

  • last (bool) – Replacing the last message

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

Tuple with two np.ndarray: 1. np.ndarray with hand-crafted features 2. np.ndarray with deep features

Return type:

Tuple[np.ndarray, np.ndarray]

get_text_features(path: str, asr: bool = False, lang: str = 'ru', show_text: bool = False, out: bool = True, runtime: bool = True, run: bool = True)[source]

Extracting features from text

Parameters:

  • path (str) – Path to video file or text

  • asr (bool) – Automatic speech recognition

  • lang (str) – Language

  • show_text (bool) – Text display

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

Tuple with two np.ndarray: 1. np.ndarray with hand-crafted features 2. np.ndarray with deep features

Return type:

Tuple[np.ndarray, np.ndarray]

get_text_union_predictions(depth: int = 1, recursive: bool = False, asr: bool = False, lang: str = 'ru', accuracy=True, url_accuracy: str = '', logs: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Obtaining text scores

Parameters:

  • depth (int) – Hierarchy depth for getting data

  • recursive (bool) – Recursive data search

  • asr (bool) – Automatic speech recognition

  • lang (str) – Language

  • accuracy (bool) – Accuracy calculation

  • url_accuracy (str) – Full path to the file with ground truth scores for calculating accuracy

  • logs (bool) – If necessary, generate a LOG file

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if scores are successfully received, otherwise False

Return type:

bool

load_text_features(url: str = None, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Downloading the dictionary with hand-crafted features

Parameters:

  • url (str) – Полный путь к лингвистическому словарю

  • force_reload (bool) – Forced download of a file with weights of a neural network model from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the hand-crafted feature dictionary is downloaded, otherwise False

Return type:

bool

load_text_model_b5(show_summary: bool = False, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Formation of a neural network model architecture for the personality traits scores

Parameters:

  • show_summary (bool) – Displaying the formed neural network architecture of the model

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the neural network architecture of the model is formed, otherwise False

Return type:

bool

load_text_model_hc(corpus: str = '', show_summary: bool = False, out: bool = True, runtime: bool = True, run: bool = True)[source]

Formation of the neural network architecture of the model for obtaining scores by hand-crafted features

Parameters:

  • corpus (str) – A corpus for testing a neural network model

  • show_summary (bool) – Displaying the formed neural network architecture of the model

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the neural network architecture of the model is formed, otherwise False

Return type:

bool

load_text_model_nn(corpus: str = '', show_summary: bool = False, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Formation of a neural network architecture for obtaining scores by deep features

Parameters:

  • corpus (str) – A corpus for testing a neural network model

  • show_summary (bool) – Displaying the formed neural network architecture of the model

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the neural network architecture of the model is formed, otherwise False

Return type:

bool

load_text_model_weights_b5(url: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Downloading the weights of a neural network model to obtain the personality traits scores

Parameters:

  • url (str) – Full path to the file with weights of the neural network model

  • force_reload (bool) – Forced download of files with weights of neural network models from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the weights of the neural network model are downloaded, otherwise False

Return type:

bool

load_text_model_weights_hc(url: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Downloading the weights of the neural network model to obtain scores by hand-crafted features

Parameters:

  • url (str) – Full path to the file with weights of the neural network model

  • force_reload (bool) – Forced download of a file with weights of a neural network model from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the weights of the neural network model are downloaded, otherwise False

Return type:

bool

load_text_model_weights_nn(url: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Downloading the weights of the neural network model to obtain scores for deep features

Parameters:

  • url (str) – Full path to the file with weights of the neural network model

  • force_reload (bool) – Forced download of a file with weights of a neural network model from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the weights of the neural network model are downloaded, otherwise False

Return type:

bool

setup_bert_encoder(url: str = None, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Formation of tokenizer and BERT model

Parameters:

  • url (str) – Полный путь к файлу с нейросетевой моделью BERT

  • force_reload (bool) – Forced downloading of a file with the BERT model from the network

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the tokenizer and the BERT model are formed, otherwise False

Return type:

bool

setup_translation_model(url: str = None, out: bool = True, runtime: bool = True, run: bool = True) → bool[source]

Formation of a tokenizer and neural network model of machine translation

Parameters:

  • url (str) – Полный путь к файлу с моделью для перевода языка

  • out (bool) – Display

  • runtime (bool) – Run time calculation

  • run (bool) – Run blocking

Returns:

True if the tokenizer and neural network model are generated, otherwise False

Return type:

bool

property text_model_b5_: Module | None

Получение нейросетевой модели nn.Module для получения оценок персональных качеств

Returns:

Нейросетевая модель nn.Module или None

Return type:

Optional[nn.Module]

property text_model_hc_: Module | None

Получение нейросетевой модели nn.Module для получения оценок по экспертным признакам

Returns:

Нейросетевая модель nn.Module или None

Return type:

Optional[nn.Module]

property text_model_nn_: Module | None

Получение нейросетевой модели nn.Module для получения оценок по нейросетевым признакам

Returns:

Нейросетевая модель tnn.Module или None

Return type:

Optional[nn.Module]