Speak2Label: using domain knowledge for creating a large scale driver gaze zone estimation dataset

Shreya Ghosh, Abhinav Dhall, Garima Sharma, Sarthak Gupta, Nicu Sebe

Research output: Chapter in Book/Report/Conference proceedingConference PaperResearchpeer-review

12 Citations (Scopus)


Labelling of human behavior analysis data is a complex and time consuming task. In this paper, a fully automatic technique for labelling an image based gaze behavior dataset for driver gaze zone estimation is proposed. Domain knowledge is added to the data recording paradigm and later labels are generated in an automatic manner using Speech To Text conversion (STT). In order to remove the noise in the STT process due to different illumination and ethnicity of subjects in our data, the speech frequency and energy are analysed. The resultant Driver Gaze in the Wild (DGW) dataset contains 586 recordings, captured during different times of the day including evenings. The large scale dataset contains 338 subjects with an age range of 18-63 years. As the data is recorded in different lighting conditions, an illumination robust layer is proposed in the Convolutional Neural Network (CNN). The extensive experiments show the variance in the dataset resembling real-world conditions and the effectiveness of the proposed CNN pipeline. The proposed network is also fine-tuned for the eye gaze prediction task, which shows the discriminativeness of the representation learnt by our network on the proposed DGW dataset. Project Page: https://sites.google.com/view/drivergazeprediction/home

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021
EditorsDima Damen, Tal Hassner, Chris Pal, Yoichi Sato
Place of PublicationPiscataway NJ USA
PublisherIEEE, Institute of Electrical and Electronics Engineers
Number of pages10
ISBN (Electronic)9781665401913
ISBN (Print)9781665401920
Publication statusPublished - 2021
EventIEEE/CVF International Conference on Computer Vision Workshops 2021 - Online, Canada
Duration: 11 Oct 202117 Oct 2021
Conference number: 18th
https://ieeexplore.ieee.org/xpl/conhome/9607382/proceeding (Proceedings)

Publication series

NameProceedings of the IEEE International Conference on Computer Vision
PublisherIEEE, Institute of Electrical and Electronics Engineers
ISSN (Print)1550-5499
ISSN (Electronic)2473-9944


ConferenceIEEE/CVF International Conference on Computer Vision Workshops 2021
Abbreviated titleICCVW 2021
Internet address

Cite this