A picture is worth a thousand words. Visual representation is one of the dominant forms of social media. The emotions that viewers feel when observing a visual content is often referred to as the content's visual sentiment. Analysis of visual sentiment has become increasingly important due to the huge volume of online visual data generated by users of social media. Automatic assessment of visual sentiment has many applications, such as monitoring the mood of the population in social media platforms (e.g., Twitter, Facebook), facilitating advertising, and understanding user behavior. However, in contrast to the extensive research on predicting textual sentiment, relatively less work has been done on sentiment analysis of visual content. In contrast to textual sentiment, visual sentiment is more subjective and implicit. There exists significant semantic gap between high-level visual perception and low-level computational attributes.
Our research explores both image and cognitive factors that influence human perception of visual sentiment through a series of psychophysics experiments. Based on such understanding of human perception of visual sentiment, we are able to model visual sentiment both empirically and computationally. Our goal is to build a visual sentiment predictor that is consistent with human perception, as a first step to enable computers to perceive emotions like humans do.
S. Fan*, Z. Shen*, M. Jiang, B. Koenig, M. Kankanhalli, Q.Zhao. Emotional attention: From eye tracking to computational modeling, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), Feb 2023.
[code & dataset]
[pdf]
[supplementary]
A. Singh, S. Fan, M. Kankanhalli, "Human Attributes Prediction Under Privacy-preserving Conditions". ACM Multimedia Conference (ACMMM) 2021. (Oral).
[code]
[dataset]
[pdf]
S. Fan, Z. Shen, B. Koenig, T. -T Ng and M. Kankanhalli. "When and Why Static Images Are More Effective Than Videos." IEEE Transactions on Affective Computing 01 (2020): 1-1.
[code]
[code (frame extraction)]
[dataset]
[pdf]
M. Cordel, S. Fan, Z. Shen and M. Kankanhalli, Emotion-Aware Human Attention Prediction. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[pdf] [supplementary].
S. Fan, B. Koenig, Q. Zhao, M. Kankanhali, A Deeper Look at Human Visual Perception of Images. Submitted manuscript.
[dataset]
S. Fan, Z. Shen, M. Jiang, B. Koenig, J. Xu, M. Kankanhali, Q.Zhao, "Emotional Attention: A Study of Image Sentiment and Visual Attention ", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 (Spotlight oral, acceptance rate: 6.6%).
[pdf]
[supplementary]
[code]
[dataset]
Project Website
S. Fan, T. -T. Ng, B. Koenig, J. Herberg, M. Jiang, Z. Shen, Q. Zhao. “Image Visual Realism: From Human Perception to Machine Computation”, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 40.9(2018).
[pdf]
[supplementary material]
[code]
[dataset]
S. Fan, M. Jiang, B. Koenig, Z. Shen, M. Kankanhali, Q.Zhao. "The Role of Visual Attention in Sentiment Prediction", ACM Multimedia, Mountain View, CA, USA, 2017.
[pdf]
[supplementary material]
[video]
[code]
[dataset]
S. Fan, M. Jiang, B. Koenig, J. Xu, Y. Chen, M. Kankanhalli, Q. Zhao. "A Correlational Study Between Human Attention and High-level Image Perception". Vision Science Society, 2017. [pdf]
S. Fan, T.-T. Ng, B. Koenig, M. Jiang, Q.Zhao. A Paradigm for Building Generalized Models of Human Image Perception through Data Fusion, CVPR 2016. [pdf]
B. Xue, M. Jiang, S. Fan, Q. Zhao. Data fusion with Deep Boltzmann Machines for high-level image perception, WICV 2016, in conjunction with CVPR 2016. [pdf]
S. Fan, T.-T. Ng, J. Herbert, B. Koenig, C. Tan, R. Wang, “Human Perception of Visual Realism for Photo and Computer-generated Face Images", ACM Transaction on Applied Perception (TAP), 11.2 (2014): 7. [pdf(12M)] [downsized pdf(300K)] [poster]
S. Fan, T.-T. Ng, J. Herbert, B. Koenig, C. Tan, R. Wang, “An Automated Estimator of Image Visual Realism Based on Human Cognition”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014. [pdf(19M)] [downsized pdf(400K)] [supplementary material] [video] [poster]
S. Fan, T.-T. Ng, J. Herbert, B. Koenig, X. Shi, “Real or Fake? Human judgments about photographs and computer-generated images of faces”, Technical Brief, ACM Siggraph Asia, Nov 2012. [pdf] [video]