Multimodal First Impression Analysis with Deep Residual Networks

Yağmur Güçlütürk (1) , Umut Güçlü (1) , Xavier Baró (2) , Hugo Jair Escalante (3) , Isabelle Guyon (4) , Sergio Escalera (2) , Marcel A. J. van Gerven (1) , Rob van Lier (1)

1 Donders Institute for Brain, Cognition and Behaviour
2 CVC - Computer Vision Center (Centre de visio per computador)
3 INAOE - Instituto Nacional de Astrofísica, Óptica y Electrónica
4 TAU - TAckling the Underspecified

People form first impressions about the personalities of unfamiliar individuals even after very brief interactions with them. In this study we present and evaluate several models that mimic this automatic social behavior. Specifically, we present several models trained on a large dataset of short YouTube video blog posts for predicting apparent Big Five personality traits of people and whether they seem suitable to be recommended to a job interview. Along with presenting our audiovisual approach and results that won the third place in the ChaLearn First Impressions Challenge, we investigate modeling in different modalities including audio only, visual only, language only, audiovisual, and combination of audiovisual and language. Our results demonstrate that the best performance could be obtained using a fusion of all data modalities.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Réseau de neurones [cs.NE]; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]

Yağmur Güçlütürk, Umut Güçlü, Xavier Baró, Hugo Jair Escalante, Isabelle Guyon, et al.. Multimodal First Impression Analysis with Deep Residual Networks. IEEE Transactions on Affective Computing, 2017, PP (99), pp.1-14. ⟨10.1109/TAFFC.2017.2751469⟩. ⟨hal-01668375⟩