NDLI: Naming multi-modal clusters to identify persons in TV broadcast

Content Provider	Springer Nature Link
Author	Poignant, Johann Fortier, Guillaume Besacier, Laurent Quét, Georges
Copyright Year	2015
Abstract	Persons’ identification in TV broadcast is one of the main tools to index this type of videos. The classical way is to use biometric face and speaker models, but, to cover a decent number of persons, costly annotations are needed. Over the recent years, several works have proposed to use other sources of names for identifying people, such as pronounced names and written names. The main idea is to form face/speaker clusters based on their similarities and to propagate these names onto clusters. In this paper, we propose a method to take advantage of written names during the diarization process, in order to both name clusters and prevent the fusion of two clusters named differently. First, we extract written names with the LOOV tool (Poignant et al. 2012); these names are associated to their co-occurring speaker turns / face tracks. Simultaneously, we build a multi-modal matrix of distances between speaker turns and face tracks. Then agglomerative clustering is performed on this matrix with the constraint to avoid merging clusters associated to different names. We also integrate the prediction of few biometric models (anchors, some journalists) to directly identify speaker turns / face tracks before the clustering process. Our approach was evaluated on the REPERE corpus and reached an F-measure of 68.2 % for speaker identification and 60.2 % for face identification. Adding few biometric models improves results and leads to 82.4 % and 65.6 % for speaker and face identity respectively. By comparison, a mono-modal, supervised person identification system with 706 speaker models trained on matching development data and additional TV and radio data provides 67.8 % F-measure, while 908 face models provide only 30.5 % F-measure.
Starting Page	8999
Ending Page	9023
Page Count	25
File Format	PDF
ISSN	13807501
Journal	Multimedia Tools and Applications
Volume Number	75
Issue Number	15
e-ISSN	15737721
Language	English
Publisher	Springer US
Publisher Date	2015-07-01
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Multimodal fusion VideoOCR Face and speaker identification TV broadcast Multimedia Information Systems Computer Communication Networks Data Structures, Cryptology and Information Theory Special Purpose and Application-Based Systems
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software Media Technology

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Lexical speaker identification in TV shows

Multimodal information fusion application to human emotion recognition from face and speech

Multimodal concept detection in broadcast media: KavTan

Enhanced long-range personal identification based on multimodal information of human features

Multimodal speaker clustering in full length movies

Person re-identification in TV series using robust face recognition and user feedback

A one-shot domain-independent robust multimedia clustering methodology based on hybrid multimodal fusion

Editorial Note: Multimodal Data Fusion, Learning and Applications

Can feature information interaction help for information fusion in multimedia problems?

Naming multi-modal clusters to identify persons in TV broadcast

Similar Documents

Lexical speaker identification in TV shows

Multimodal information fusion application to human emotion recognition from face and speech

Multimodal concept detection in broadcast media: KavTan

Enhanced long-range personal identification based on multimodal information of human features

Multimodal speaker clustering in full length movies

Person re-identification in TV series using robust face recognition and user feedback

A one-shot domain-independent robust multimedia clustering methodology based on hybrid multimodal fusion

Editorial Note: Multimodal Data Fusion, Learning and Applications

Can feature information interaction help for information fusion in multimedia problems?

Naming multi-modal clusters to identify persons in TV broadcast