Please use this identifier to cite or link to this item:
Appears in Collections:Computing Science and Mathematics Journal Articles
Peer Review Status: Refereed
Title: Towards an intelligent framework for multimodal affective data analysis
Author(s): Poria, Soujanya
Cambria, Erik
Hussain, Amir
Huang, Guang-Bin
Contact Email:
Keywords: Multimodal
Multimodal sentiment analysis
Facial expressions
Emotion analysis
Affective computing
Issue Date: Mar-2015
Date Deposited: 10-Dec-2014
Citation: Poria S, Cambria E, Hussain A & Huang G (2015) Towards an intelligent framework for multimodal affective data analysis. Neural Networks, 63, pp. 104-116.
Abstract: An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook everyday. In order to cope with the growth of such so much multimodal data, there is an urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this paper, we propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user-generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human-computer interaction. In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video) features to enhance the multimodal information extraction process. In preliminary experiments using the eNTERFACE dataset, our proposed multi-modal system is shown to achieve an accuracy of 87.95%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 56% reduction in error rate.
DOI Link: 10.1016/j.neunet.2014.10.005
Rights: Published in Neural Networks by Elsevier; Elsevier believes that individual authors should be able to distribute their AAMs for their personal voluntary needs and interests, e.g. posting to their websites or their institution’s repository, e-mailing to colleagues. However, our policies differ regarding the systematic aggregation or distribution of AAMs to ensure the sustainability of the journals to which AAMs are submitted. Therefore, deposit in, or posting to, subject-oriented or centralized repositories (such as PubMed Central), or institutional repositories with systematic posting mandates is permitted only under specific agreements between Elsevier and the repository, agency or institution, and only consistent with the publisher’s policies concerning such repositories. Voluntary posting of AAMs in the arXiv subject repository is permitted.

Files in This Item:
File Description SizeFormat 
Neural Networks 2014.pdfFulltext - Accepted Version3.23 MBAdobe PDFView/Open

This item is protected by original copyright

Items in the Repository are protected by copyright, with all rights reserved, unless otherwise indicated.

The metadata of the records in the Repository are available under the CC0 public domain dedication: No Rights Reserved

If you believe that any material held in STORRE infringes copyright, please contact providing details and we will remove the Work from public display in STORRE and investigate your claim.