Research

How do humans perceive & navigate their social world?

Social Perception • Serial Dependence • Bayesian Models • Visual Working Memory

The world around us is incredibly rich and complex. Humans navigate through this complex world while processing social information, like others’ emotions, goals, and actions, in a fraction of a second. How do humans process such complex information so quickly and how can we build computational and computer vision models to process this information as rapidly and efficiently as humans do intrinsically? My research aims to understand how humans process complex social information in real-time by using more naturalistic stimuli like movies and videos. I use a range of tools like electroencephalograms (EEG) , eye-tracking, computational models, and psychophysics to investigate these questions.

Doctoral Research

VEATIC: Video-based Emotion and Affect Tracking in Context Dataset

Recognizing the emotion of others is incredibly important for navigating and understanding the social world around us. Consequently, many AI models have been developed with the goal of automatically perceiving and interpreting human emotions.

Multiple datasets have been created with the goal of training computer vision models to infer the emotions of humans. However, many of these datasets lack diversity in terms of characters, motion, and context! Context can be a very important clue in emotion perception (Figure 1).

In our work, we introduce a brand new large dataset that can be beneficial to both psychology and computer science groups: the Video-based Emotion and Affect Tracking in Context Dataset (VEATIC)!

VEATIC has 124 videos collected from multiple media sources with continuous valence and arousal ratings of each frame via real-time annotation from 194 annotators. The distribution of affective space in VEATIC includes a wide range of valence/arousal ratings as well as many unique contexts (Figure 2).

Reference:

†Ren, Z., †Ortega, J., †Wang, Y., Chen, Z., Whitney, D., Guo, Y., & Yu, S. X. (2023). VEATIC: Video-based Emotion and Affect Tracking in Context Dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4467-4477).

Figure 1. The importance of context in emotion recognition.

Figure 2. Distribution of valence and arousal ratings across videos in the VEATIC dataset.

Impaired context-based emotion processing in individuals with high Autism Quotient scores

Emotion perception is vital for successful social interactions. However, individuals with autism spectrum disorder (ASD) experience social communication deficits and have reported difficulties in facial expression recognition.

However, emotion recognition depends on more than just processing facial expressions; context is critically important to correctly infer the emotions of others. Whether context-based emotion processing is impacted in those with Autism remains unclear.

In our study, we used a recently developed context-based emotion perception task, called Inferential Emotion Tracking (IET), and investigated whether individuals who scored high on the Autism Spectrum Quotient (AQ) had deficits in context-based emotion perception.

We found that individual differences in Autism Quotient scores were more strongly correlated with IET task accuracy than they are with traditional face emotion perception tasks. This correlation remained significant even when controlling for potential covarying factors, general intelligence, and performance on traditional face perception tasks.

Our results bring into focus a range of previous mixed findings on the relationship between emotion perception and ASD, and they shed light on possible avenues for assessing and treating ASD in future work.

Reference:

Ortega, J., Chen, Z., & Whitney, D. (2023). Inferential Emotion Tracking reveals impaired context-based emotion processing in individuals with high Autism Quotient scores. Scientific Reports, 13(1), 8093.

Figure 1. Correlations between IET and questionnaires.

Figure 2. Significance tests for AQ correlations. (a) Partial correlations, (b) permutation tests.

Serial Dependence in Emotion Perception

The visual world around is naturally autocorrelated meaning that visual information that we currently see is similar to information we have seen seconds ago. These correlations are an example of dynamic natural scene statistics that we encounter in our visual experience.

We experience these same autocorrelations in emotion, a form of natural emotion statistics, where we can perceive a person smiling and predict the future time-course of their emotion. But, does the human visual system take into account these natural emotion statistics when recognizing displays of affect and emotion?

In our study, we quantified the natural emotion statistics in videos by measuring the autocorrelations in emotional content present in films and movies. The results showed that observers’ perception of emotion was smoothed over ∼12 seconds or more. This time-course closely followed the temporal fluctuations in visual information about emotion found in natural scenes. Moreover, the temporal and feature tuning of the perceptual smoothing was consistent with known properties of serial dependence.

Our findings suggest that serial dependence is introduced in the perception of emotion to match the natural autocorrelations that are observed in the real world, an operation that could improve the efficiency, sensitivity, and stability of emotion perception.

Reference:

Ortega, J., Chen, Z., & Whitney, D. (2023). Serial dependence in emotion perception mirrors the autocorrelations in natural emotion statistics. Journal of Vision, 23(3), 12.

Figure 1. Group autocorrelations for the perceptual and movie-based autocorrelation.

Figure 2. Serial dependence in the perception of emotion and effect sizes.

Check back often for updates!

Undergraduate Research

Eye blink rate and Working Memory

The healthy human blinks around 15-20 times per minute, but the precorneal tear film, which lubricates the eye, only begins drying up approximately 25 seconds after a blink ends. This means that humans blink more often than needed.

Previous work investigating spontaneous eye blink rate has found that our blink rates can change depending on the task we are currently doing. For example, reading is accompanied by low levels of spontaneous eye blink rate, while high levels of spontaneous eye blink rate have been reported when individuals are having a conversation.

So, why do we blink? During my undergraduate studies, I investigated this question by looking at how spontaneous eye blink rate changed during different phases of a working memory task. We found, across two studies, that during the Delay period of the task, which is when participants have to maintain information in working memory, spontaneous eye blink rate was correlated with working memory performance.

Our findings provide further evidence suggests that pontaneous eye blink rate could be an important predictor of working memory task performance with the changes during the delay period suggesting a role in working memory maintenance.

Reference:

Ortega, J., Plaska, C. R., Gomes, B. A., & Ellmore, T. M. (2022). Spontaneous Eye Blink Rate During the Working Memory Delay Period Predicts Task Accuracy. Frontiers in Psychology, 13, 788231.

Correlation between sEBR during different phases of the working memory task and task accuracy. Correlation plots show sEBR on the x-axis and task accuracy on the y-axis.

sEBR modulation across working memory task periods.