Service interruption on Monday 11 July from 12:30 to 13:00: all the sites of the CCSD (HAL, EpiSciences, SciencesConf, AureHAL) will be inaccessible (network hardware connection).
Skip to Main content Skip to Navigation
Journal articles

LAEO-Net++: revisiting people Looking At Each Other in videos

Abstract : Capturing the 'mutual gaze' of people is essential for understanding and interpreting the social interactions between them. To this end, this paper addresses the problem of detecting people Looking At Each Other (LAEO) in video sequences. For this purpose, we propose LAEO-Net++, a new deep CNN for determining LAEO in videos. In contrast to previous works, LAEO-Net++ takes spatio-temporal tracks as input and reasons about the whole track. It consists of three branches, one for each character's tracked head and one for their relative position. Moreover, we introduce two new LAEO datasets: UCO-LAEO and AVA-LAEO. A thorough experimental evaluation demonstrates the ability of LAEO-Net++ to successfully determine if two people are LAEO and the temporal window where it happens. Our model achieves state-of-the-art results on the existing TVHID-LAEO video dataset, significantly outperforming previous approaches. Finally, we apply LAEO-Net++ to a social network, where we automatically infer the social relationship between pairs of people based on the frequency and duration that they LAEO, and show that LAEO can be a useful tool for guided search of human interactions in videos. The code is available at
Complete list of metadata
Contributor : Vicky Kalogeiton Connect in order to contact the contributor
Submitted on : Thursday, January 14, 2021 - 4:44:18 PM
Last modification on : Friday, January 22, 2021 - 3:30:54 AM
Long-term archiving on: : Thursday, April 15, 2021 - 7:17:59 PM


Files produced by the author(s)




Manuel J Marín-Jiménez, Vicky Kalogeiton, Pablo Medina-Suárez, Andrew Zisserman. LAEO-Net++: revisiting people Looking At Each Other in videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2020, pp.1-16. ⟨10.1109/TPAMI.2020.3048482⟩. ⟨hal-03110590⟩



Record views


Files downloads