Multimodal information fusion

class oceanai.modules.lab.prediction.PredictionMessages(lang: str = 'ru', color_simple: str = '#666', color_info: str = '#1776D2', color_err: str = '#FF0000', color_true: str = '#008001', bold_text: bool = True, text_runtime: str = '', num_to_df_display: int = 30)

Bases: Audio, Video, Text

Class for messages

Parameters:

• lang (str) – See lang

• color_simple (str) – See color_simple

• color_info (str) – See color_info

• color_err (str) – See color_err

• color_true (str) – See color_true

• bold_text (bool) – See bold_text

• num_to_df_display (int) – See num_to_df_display

• text_runtime (str) – See text_runtime

class oceanai.modules.lab.prediction.Prediction(lang: str = 'ru', color_simple: str = '#666', color_info: str = '#1776D2', color_err: str = '#FF0000', color_true: str = '#008001', bold_text: bool = True, text_runtime: str = '', num_to_df_display: int = 30)

Bases: PredictionMessages

Class for multimodal information fusion

Parameters:

• lang (str) – See lang

• color_simple (str) – See color_simple

• color_info (str) – See color_info

• color_err (str) – See color_err

• color_true (str) – See color_true

• bold_text (bool) – See bold_text

• num_to_df_display (int) – See num_to_df_display

• text_runtime (str) – See text_runtime

__concat_pred_av(pred_hc_audio: ndarray, pred_nn_audio: ndarray, pred_hc_video: ndarray, pred_nn_video: ndarray, out: bool = True) → List[ndarray | None]

Concatenation of scores by hand-crafted and deep features (multimodal)

Note

private method

Parameters:

• pred_hc_audio (np.ndarray) – Normalized scores by hand-crafted features (audio modality)

• pred_nn_audio (np.ndarray) – Scores based on deep features (audio modality)

• pred_hc_video (np.ndarray) – Scores based on hand-crafted features (video modality)

• pred_nn_video (np.ndarray) – Scores based on deep features (video modality)

• out (bool) – Display

Returns:

Concatenated scores by hand-crafted and deep features

Return type:

List[Optional[np.ndarray]]

Examples

True – 1 –

In [1]:

import numpy as np
from oceanai.modules.lab.prediction import Prediction

pred = Prediction()

arr_hc_audio = np.array([
    [0.64113516, 0.6217892, 0.54451424, 0.6144415, 0.59334993],
    [0.6652424, 0.63606125, 0.572305, 0.63169795, 0.612515]
])

arr_nn_audio = np.array([
    [0.56030345, 0.7488746, 0.44648764, 0.59893465, 0.5701077],
    [0.5900006, 0.7652722, 0.4795154, 0.6409055, 0.6088242]
])
arr_hc_video = np.array([
    [0.67113516, 0.6517892, 0.59451424, 0.6344415, 0.53334993],
    [0.6852424, 0.62606125, 0.562305, 0.67169795, 0.672515]
])

arr_nn_video = np.array([
    [0.58030345, 0.7788746, 0.47648764, 0.53893465, 0.5901077],
    [0.5100006, 0.7452722, 0.4495154, 0.6909055, 0.6488242]
])

pred._Prediction__concat_pred_av(
    pred_hc_audio = arr_hc_audio,
    pred_nn_audio = arr_nn_audio,
    pred_hc_video = arr_hc_video,
    pred_nn_video = arr_nn_video,
    out = True
)

__load_av_model_b5(show_summary: bool = False, out: bool = True) → Module | None

Formation of the neural network architecture of the model to obtain the personality traits scores

Note

private method

Parameters:

• show_summary (bool) – Displaying the formed neural network architecture of the model

• out (bool) – Display

Returns:

None если неверные типы или значения аргументов, в обратном случае нейросетевая модель nn.Module для получения результата оценки персонального качества

Return type:

Optional[nn.Module]

Examples

True – 1 –

In [1]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction()

pred._Prediction__load_av_model_b5(
    show_summary = True, out = True
)

[1]:

Model: "model"
_________________________________________________________________
Layer (type)                Output Shape              Param #
=================================================================
input_1 (InputLayer)        [(None, 64)]              0

dense_1 (Dense)             (None, 1)                 65

activ_1 (Activation)        (None, 1)                 0

=================================================================
Total params: 65
Trainable params: 65
Non-trainable params: 0
_________________________________________________________________
<tf.keras.Model at 0x147892ee0>

Error – 1 –

In [2]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred._Prediction__load_av_model_b5(
    show_summary = True, out = []
)

[3]:

[2022-10-17 10:53:03] Invalid argument types or values in "Prediction.__load_av_model_b5" ...

__load_avt_model_b5(show_summary: bool = False, out: bool = True) → Module | None

Formation of the neural network architecture of the model to obtain the personality traits scores

Note

private method

Parameters:

• show_summary (bool) – Displaying the formed neural network architecture of the model

• out (bool) – Display

Returns:

None если неверные типы или значения аргументов, в обратном случае нейросетевая модель nn.Module для получения оценок персональных качеств

Return type:

Optional[nn.Module]

__load_model_weights(url: str, force_reload: bool = True, info_text: str = '', out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the weights of the neural network model

Note

private method

Parameters:

• url (str) – Full path to the file with the weights of the neural network model (non-neuroticism)

• force_reload (bool) – Forced download of files with weights of neural network models from the network

• info_text (str) – Text for informational message

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True if the weights of the neural network models are downloaded, otherwise False

Return type:

bool

__norm_pred(pred_data: ndarray, len_nn: int = 16, out: bool = True) → ndarray

Normalization of scores by hand-crafted and deep features (multimodal)

Note

private method

Parameters:

• pred_data (np.ndarray) – Scores

• len_nn (int) – The maximum size of the scores vector

• out (bool) – Display

Returns:

Normalized scores by hand-crafted and deep features (multimodal)

Return type:

np.ndarray

Examples

True – 1 –

In [1]:

import numpy as np
from oceanai.modules.lab.prediction import Prediction

pred = Prediction()

arr = np.array([
    [0.64113516, 0.6217892, 0.54451424, 0.6144415, 0.59334993],
    [0.6652424, 0.63606125, 0.572305, 0.63169795, 0.612515]
])

pred._Prediction__norm_pred(
    pred_data = arr,
    len_nn = 4,
    out = True
)

[1]:

array([
    [0.64113516, 0.6217892 , 0.54451424, 0.6144415 , 0.59334993],
    [0.6652424 , 0.63606125, 0.572305  , 0.63169795, 0.612515],
    [0.65318878, 0.62892523, 0.55840962, 0.62306972, 0.60293247],
    [0.65318878, 0.62892523, 0.55840962, 0.62306972, 0.60293247]
])

Error – 1 –

In [2]:

import numpy as np
from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

arr = np.array([])

pred._Prediction__norm_pred(
    pred_data = arr,
    len_nn = 4,
    out = True
)

[3]:

[2022-10-20 22:03:17] Invalid argument types or values in "Prediction.__norm_pred" ...

array([], dtype=float64)

property av_models_b5_: Dict[str, Module | None]

Получение нейросетевых моделей nn.Module для получения результатов оценки персональных качеств

Returns:

Словарь с нейросетевыми моделями nn.Module

Return type:

Dict

Examples

True – 1 –

In [1]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.load_av_models_b5(
    show_summary = False, out = True,
    runtime = True, run = True
)

pred.av_models_b5_

[1]:

[2022-12-08 15:21:22] Formation of neural network architectures of models for obtaining the personality traits scores (multimodal fusion) ...

--- Runtime: 0.305 sec. ---

{
    'openness': <keras.engine.functional.Functional at 0x14eee5790>,
    'conscientiousness': <keras.engine.functional.Functional at 0x14f2d9d00>,
    'extraversion': <keras.engine.functional.Functional at 0x14f2fb190>,
    'agreeableness': <keras.engine.functional.Functional at 0x14f2c7fd0>,
    'non_neuroticism': <keras.engine.functional.Functional at 0x14f2ef940>
}

Error – 1 –

In [2]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction()

pred.av_models_b5_

[2]:

{
    'openness': None,
    'conscientiousness': None,
    'extraversion': None,
    'agreeableness': None,
    'non_neuroticism': None
}

property avt_model_b5_: Module | None

Получение нейросетевой модели nn.Module для получения оценок персональных качеств

Returns:

Нейроаетевая модель nn.Module

Return type:

Dict

get_av_union_predictions(depth: int = 1, recursive: bool = False, sr: int = 44100, window_audio: int | float = 2.0, step_audio: int | float = 1.0, reduction_fps: int = 5, window_video: int = 10, step_video: int = 5, lang: str = 'ru', accuracy: bool = True, url_accuracy: str = '', logs: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Getting audio and video scores (multimodal fusion)

Parameters:

• depth (int) – Hierarchy depth for getting data

• recursive (bool) – Recursive data search

• sr (int) – Sampling frequency

• window_audio (Union[int, float]) – Audio segment window size (in seconds)

• step_audio (Union[int, float]) – Audio segment window shift step (in seconds)

• reduction_fps (int) – Frame rate reduction

• window_video (int) – Video segment window size (in frames)

• step_video (int) – Video segment window shift step (frames)

• lang (str) – Language

• accuracy (bool) – Accuracy

• url_accuracy (str) – Full path to the file with ground truth scores for accuracy

• logs (bool) – If necessary, generate a LOG file

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True if scores are successfully received, otherwise False

Return type:

bool

Example

get_avt_predictions(depth: int = 1, recursive: bool = False, sr: int = 44100, window_audio: int | float = 2.0, step_audio: int | float = 1.0, reduction_fps: int = 5, window_video: int = 10, step_video: int = 5, asr: bool = False, lang: str = 'ru', accuracy=True, url_accuracy: str = '', logs: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Getting audio, video and text scores (multimodal fusion)

Parameters:

• depth (int) – Hierarchy depth for getting data

• recursive (bool) – Recursive data search

• sr (int) – Sampling frequency

• window_audio (Union[int, float]) – Audio segment window size (in seconds)

• step_audio (Union[int, float]) – Audio segment window shift step (in seconds)

• reduction_fps (int) – Frame rate reduction

• window_video (int) – Video segment window size (in frames)

• step_video (int) – Video segment window shift step (frames)

• asr (bool) – Automatic speech recognition

• lang (str) – Language

• accuracy (bool) – Accuracy

• url_accuracy (str) – Full path to the file with ground truth scores for accuracy

• logs (bool) – If necessary, generate a LOG file

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True if scores are successfully received, otherwise False

Return type:

bool

get_avt_predictions_gradio(paths: list[str] = [], depth: int = 1, recursive: bool = False, sr: int = 44100, window_audio: int | float = 2.0, step_audio: int | float = 1.0, reduction_fps: int = 5, window_video: int = 10, step_video: int = 5, asr: bool = False, lang: str = 'ru', accuracy=True, url_accuracy: str = '', logs: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Getting audio, video and text scores (multimodal fusion)

Parameters:

• depth (int) – Hierarchy depth for getting data

• recursive (bool) – Recursive data search

• sr (int) – Sampling frequency

• window_audio (Union[int, float]) – Audio segment window size (in seconds)

• step_audio (Union[int, float]) – Audio segment window shift step (in seconds)

• reduction_fps (int) – Frame rate reduction

• window_video (int) – Video segment window size (in frames)

• step_video (int) – Video segment window shift step (frames)

• asr (bool) – Automatic speech recognition

• lang (str) – Language

• accuracy (bool) – Accuracy

• url_accuracy (str) – Full path to the file with ground truth scores for accuracy

• logs (bool) – If necessary, generate a LOG file

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

• paths (list[str])

Returns:

True if scores are successfully received, otherwise False

Return type:

bool

load_av_models_b5(show_summary: bool = False, out: bool = True, runtime: bool = True, run: bool = True) → bool

Formation of neural network architectures of models for obtaining the personality traits scores

Parameters:

• show_summary (bool) – Displaying the last generated neural network architecture of models

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True еif the neural network architectures of the model are formed, otherwise False

Return type:

bool

Examples

True – 1 –

In [1]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.load_av_models_b5(
    show_summary = True, out = True,
    runtime = True, run = True
)

[1]:

[2022-12-08 15:19:30] Формирование нейросетевых архитектур моделей для получения результатов оценки персональных качеств (мультимодальное объединение) ...

Model: "model_4"
_________________________________________________________________
Layer (type)                Output Shape              Param #
=================================================================
input_1 (InputLayer)        [(None, 64)]              0

dense_1 (Dense)             (None, 1)                 65

activ_1 (Activation)        (None, 1)                 0

=================================================================
Total params: 65
Trainable params: 65
Non-trainable params: 0
_________________________________________________________________
--- Время выполнения: 0.141 сек. ---

True

Error – 1 –

In [2]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.load_av_models_b5(
    show_summary = 1, out = True,
    runtime = True, run = True
)

[2]:

[2022-12-08 15:20:36] Invalid argument types or values in "Prediction.load_av_models_b5" ...

False

load_av_models_weights_b5(url_openness: str, url_conscientiousness: str, url_extraversion: str, url_agreeableness: str, url_non_neuroticism: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the weights of neural network models to obtain the personality traits scores

Parameters:

• url_openness (str) – Full path to the file with the weights of the neural network model (openness)

• url_conscientiousness (str) – Full path to the file with the weights of the neural network model (conscientiousness)

• url_extraversion (str) – Full path to the file with the weights of the neural network model (extraversion)

• url_agreeableness (str) – Full path to the file with the weights of the neural network model (agreeableness)

• url_non_neuroticism (str) – Full path to the file with the weights of the neural network model (non-neuroticism)

• force_reload (bool) – Forced download of files with weights of neural network models from the network

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True if the weights of the neural network models are downloaded, otherwise False

Return type:

bool

Examples

True – 1 –

In [1]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.load_av_models_b5(
    show_summary = False, out = True,
    runtime = True, run = True
)

[1]:

[2022-12-08 16:56:37] Formation of neural network architectures of models for obtaining the personality traits scores (multimodal fusion) ...

--- Runtime: 0.075 sec. ---

True

In [2]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.path_to_save_ = './models'
pred.chunk_size_ = 2000000

url_openness = pred.weights_for_big5_['av']['b5']['openness']['sberdisk']
url_conscientiousness = pred.weights_for_big5_['av']['b5']['conscientiousness']['sberdisk']
url_extraversion = pred.weights_for_big5_['av']['b5']['extraversion']['sberdisk']
url_agreeableness = pred.weights_for_big5_['av']['b5']['agreeableness']['sberdisk']
url_non_neuroticism = pred.weights_for_big5_['av']['b5']['non_neuroticism']['sberdisk']

pred.load_av_models_weights_b5(
    url_openness = url_openness,
    url_conscientiousness = url_conscientiousness,
    url_extraversion = url_extraversion,
    url_agreeableness = url_agreeableness,
    url_non_neuroticism = url_non_neuroticism,
    force_reload = True,
    out = True,
    runtime = True,
    run = True
)

[2]:

[2022-12-08 17:03:18] Downloading the weights of neural network models to obtain the personality traits scores (multimodal fusion) ...

[2022-12-08 17:03:21] File download "weights_2022-08-28_11-14-35.h5" (100.0%) ... Openness

[2022-12-08 17:03:21] File download "weights_2022-08-28_11-08-10.h5" (100.0%) ... Conscientiousness

[2022-12-08 17:03:21] File download "weights_2022-08-28_11-17-57.h5" (100.0%) ... Extraversion

[2022-12-08 17:03:21] File download "weights_2022-08-28_11-25-11.h5" (100.0%) ... Agreeableness

[2022-12-08 17:03:21] File download "weights_2022-06-14_21-44-09.h5" (100.0%) ... Non-Neuroticism

--- Runtime: 3.399 sec. ---

True

Error – 1 –

In [3]:

from oceanai.modules.lab.prediction import Prediction

pred = Prediction(lang = 'en')

pred.path_to_save_ = './models'
pred.chunk_size_ = 2000000

url_openness = pred.weights_for_big5_['av']['b5']['openness']['sberdisk']
url_conscientiousness = pred.weights_for_big5_['av']['b5']['conscientiousness']['sberdisk']
url_extraversion = pred.weights_for_big5_['av']['b5']['extraversion']['sberdisk']
url_agreeableness = pred.weights_for_big5_['av']['b5']['agreeableness']['sberdisk']
url_non_neuroticism = pred.weights_for_big5_['av']['b5']['non_neuroticism']['sberdisk']

pred.load_av_models_weights_b5(
    url_openness = url_openness,
    url_conscientiousness = url_conscientiousness,
    url_extraversion = url_extraversion,
    url_agreeableness = url_agreeableness,
    url_non_neuroticism = url_non_neuroticism,
    force_reload = True,
    out = True,
    runtime = True,
    run = True
)

[3]:

[2022-12-08 17:05:32] Downloading the weights of neural network models to obtain the personality traits scores (multimodal fusion) ...

[2022-12-08 17:05:32] File download "weights_2022-08-28_11-14-35.h5" (100.0%) ...

[2022-12-08 17:05:33] Something went wrong ... ailed to load neural network model weights ... Openness

    File: /Users/dl/GitHub/oceanai/oceanai/modules/lab/prediction.py
    Line: 639
    Method: load_av_models_weights_b5
    Error type: AttributeError

[2022-12-08 17:05:33] File download "weights_2022-08-28_11-08-10.h5" (100.0%) ...

[2022-12-08 17:05:33] Something went wrong ... ailed to load neural network model weights ... Conscientiousness

    File: /Users/dl/GitHub/oceanai/oceanai/modules/lab/prediction.py
    Line: 639
    Method: load_av_models_weights_b5
    Error type: AttributeError

[2022-12-08 17:05:33] File download "weights_2022-08-28_11-17-57.h5" (100.0%) ...

[2022-12-08 17:05:33] Something went wrong ... ailed to load neural network model weights ... Extraversion

    File: /Users/dl/GitHub/oceanai/oceanai/modules/lab/prediction.py
    Line: 639
    Method: load_av_models_weights_b5
    Error type: AttributeError

[2022-12-08 17:05:33] File download "weights_2022-08-28_11-25-11.h5" (100.0%) ...

[2022-12-08 17:05:33] Something went wrong ... ailed to load neural network model weights ... Agreeableness

    File: /Users/dl/GitHub/oceanai/oceanai/modules/lab/prediction.py
    Line: 639
    Method: load_av_models_weights_b5
    Error type: AttributeError

[2022-12-08 17:05:33] File download "weights_2022-06-14_21-44-09.h5" (100.0%) ...

[2022-12-08 17:05:33] Something went wrong ... ailed to load neural network model weights ... Non-Neuroticism

    File: /Users/dl/GitHub/oceanai/oceanai/modules/lab/prediction.py
    Line: 639
    Method: load_av_models_weights_b5
    Error type: AttributeError

--- Runtime: 1.024 sec. ---

False

load_avt_model_b5(show_summary: bool = False, out: bool = True, runtime: bool = True, run: bool = True) → bool

Formation of the neural network architecture of the model to obtain the personality traits scores

Parameters:

• show_summary (bool) – Displaying the last generated neural network architecture of models

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True еif the neural network architectures of the model are formed, otherwise False

Return type:

bool

load_avt_model_weights_b5(url: str, force_reload: bool = True, out: bool = True, runtime: bool = True, run: bool = True) → bool

Downloading the weights of neural network models to obtain the personality traits scores

Parameters:

• url (str) – Full path to the file with the weights of the neural network model (non-neuroticism)

• force_reload (bool) – Forced download of files with weights of neural network models from the network

• out (bool) – Display

• runtime (bool) – Runtime count

• run (bool) – Run blocking

Returns:

True if the weights of the neural network models are downloaded, otherwise False

Return type:

bool