Documentation of the Strong AI Algorithm Library - OCEAN-AI

OCEAN-AI is an open-source library consisting of a set of algorithms for intellectual analysis of human behavior based on multimodal data for automatic personality traits (PT) assessment. The library evaluates five PT: Openness to experience, Conscientiousness, Extraversion, Agreeableness, Non-Neuroticism.

PyPI (image) PyPI - Python Version (image) PyPI - Implementation (image) GitHub repo size (image) PyPI - Status (image) PyPI - License (image) GitHub top language (image)
Pipeline of the OCEAN-AI library

OCEAN-AI includes four main algorithms:

  1. Audio Information Analysis Algorithm (AIA).

  2. Video Information Analysis Algorithm (VIA).

  3. Text Information Analysis Algorithm (TIA).

  4. Multimodal Information Fusion Algorithm (MIF).

The AIA, VIA and TIA algorithms implement the functions of strong artificial intelligence (AI) in terms of complexing acoustic, visual and linguistic features built on different principles (hand-crafted and deep features), i.e. these algorithms implement the approaches of composite (hybrid) AI. The necessary pre-processing is carried out in the algorithms audio, video and text information, the calculation of acoustic, visual and linguistic features and the output of PT predictions based on them.

The MIF algorithm is a link between three information analysis algorithms (AIA, VIA and TIA). This algorithm performs a weighted neural network combination of prediction personality traits obtained using the AIA, VIA and TIA algorithms.

OCEAN-AI provides examples of solving practical tasks based on obtained PT scores:

  1. Ranking potential candidates by professional responsibilities by:

    1. professional groups;

    2. professional skills.

  2. Predicting consumer preferences for industrial goods through:

    1. the example of car characteristics;

    2. the example of mobile device application categories.

  3. Forming effective work teams:

    1. finding a suitable junior colleague;

    2. finding a suitable senior colleague.

In addition to the main task - multimodal personality traits assessment, the features implemented in OCEAN-AI features will allow researchers to solve other problems of analyzing human behavior, for example, recognizing his affective states.

OCEAN-AI uses the latest open-source libraries for audio, video and text processing: librosa, openSMILE, openCV, mediapipe, transformers.

OCEAN-AI is written in the python programming language. Neural network models are implemented and trained using an open-source library code TensorFlow.

Research data

The OCEAN-AI library was tested on two corpora:

  1. Общедоступном и крупномаштабном корпусе First Impressions V2.

  2. On the first publicly available Russian-language Multimodal Personality Traits Assessment (MuPTA) corpus.

Certificate of state registration of a computer program

Library of algorithms for intelligent analysis of human behavior based on multimodal data, providing human’s personality traits assessment to perform professional duties (OCEAN-AI)

Certificate of state registration of a database

MuPTA - Multimodal Personality Traits Assessment Corpus

Publications

Journals

@article{ryumina22_neurocomputing,
    author = {Elena Ryumina and Denis Dresvyanskiy and Alexey Karpov},
    title = {In Search of a Robust Facial Expressions Recognition Model: A Large-Scale Visual Cross-Corpus Study},
    journal = {Neurocomputing},
    volume = {514},
    pages = {435-450},
    year = {2022},
    doi = {https://doi.org/10.1016/j.neucom.2022.10.013},
}

@article{ryumina24_eswa,
    author = {Elena Ryumina and Maxim Markitantov and Dmitry Ryumin and Alexey Karpov},
    title = {OCEAN-AI Framework with EmoFormer Cross-Hemiface Attention Approach for Personality Traits Assessment},
    journal = {Expert Systems with Applications},
    volume = {239},
    pages = {122441},
    year = {2024},
    doi = {https://doi.org/10.1016/j.eswa.2023.122441},
}

Conferences

@inproceedings{ryumina23_interspeech,
    author = {Elena Ryumina and Dmitry Ryumin and Maxim Markitantov and Heysem Kaya and Alexey Karpov},
    title = {Multimodal Personality Traits Assessment (MuPTA) Corpus: The Impact of Spontaneous and Read Speech},
    year = {2023},
    booktitle = {INTERSPEECH},
    pages = {4049--4053},
    doi = {https://doi.org/10.21437/Interspeech.2023-1686},
}

Supported by The study is supported by the Research Center Strong Artificial Intelligence in Industry of ITMO University.

Strong Artificial Intelligence in industry (ITMO University)

Contents:

Indexing