NDLI: Using language structure for adaptive multimodal language acquisition

Please wait, while we are loading the content...

Two-way eye contact between humans and robots

A framework for evaluating multimodal integration by humans and a role for embodied conversational agents

Multimodal interaction in an augmented reality scenario

Walking-pad: a step-in-place locomotion interface for virtual environments

Towards integrated microplanning of language and iconic gesture for multimodal output

When do we interact multimodally?: cognitive load and multimodal communication patterns

M/ORIS: a medical/operating room interaction system

A model-based approach for real-time embedded multimodal systems in military aircrafts

Agent and library augmented shared knowledge areas (ALASKA)

A multimodal learning interface for sketch, speak and point creation of a schedule chart

Adaptations of multimodal content in dialog systems targeting heterogeneous devices

Another person's eye gaze as a cue in solving programming problems

From conversational tooltips to grounded discourse: head poseTracking in interactive dialog systems

The ThreadMill architecture for stream-oriented human communication analysis applications

Multimodal detection of human interaction events in a nursing home environment

Exploiting prosodic structuring of coverbal gesticulation

Bimodal HCI-related affect recognition

Modality fusion for graphic design applications

ICARE software components for rapidly developing multimodal interfaces

MULTIFACE: multimodal content adaptations for heterogeneous devices

Real-time audio-visual tracking for meeting analysis

Utilizing gestures to better understand dynamic structure of human communication

EyePrint: support of document browsing with eye gaze trace

Evaluation of spoken multimodal conversation

TouchLight: an imaging touch screen and display for gesture-based interaction

Elvis: situated speech and gesture understanding for a robotic chandelier

Visual and linguistic information in gesture classification

Identifying the addressee in human-human-robot interactions based on head pose and speech

Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures

MacVisSTA: a system for multimodal analysis

Command and control resource performance $predictor(C^{2}RP^{2})$

Collaboration in parallel worlds

Multimodal programming for dyslexic students

Multimodal transformed social interaction

Multimodal model integration for sentence unit detection

Articulatory features for robust visual speech recognition

AROMA: ambient awareness through olfaction in a messaging application

Context based multimodal fusion

A multi-modal architecture for cellular phones

Segmentation and classification of meetings using multiple information streams

Gestural cues for speech understanding

The virtual haptic back for palpatory training

Emotional Chinese talking head system

'SlidingMap': introducing and evaluating a new modality for map interaction

A maximum entropy based approach for multimodal integration

Using language structure for adaptive multimodal language acquisition

A vision-based sign language recognition system using tied-mixture density HMM

Experiences on haptic interfaces for visually impaired young children

Multimodal interaction for distributed collaboration

Multimodal interface platform for geographical information systems (GeoMIP) in crisis management

Private speech during multimodal human-computer interaction

Analysis of emotion recognition using facial expressions, speech and multimodal information

Visual touchpad: a two-handed gestural input device

Projection augmented models: the effect of haptic feedback on subjective and objective human factors

Support for input adaptability in the ICON toolkit

An evaluation of virtual human technology in informational kiosks

Multimodal interface design for multimodal meeting content retrieval

User walkthrough of multimodal access to multidimensional databases

Software infrastructure for multi-modal virtual environments

Determining efficient multimodal information-interaction spaces for $C^{2}$ systems

Multimodal interaction under exerted conditions in a natural field setting

GroupMedia: distributed multi-modal interfaces

Using spatial warning signals to capture a driver's visual attention

A segment-based audio-visual speech recognizer: data collection, development, and initial experiments

Multimodal interfaces and applications for visually impaired children

Multilayer architecture in sign language recognition system

Computer vision techniques and applications in human-computer interaction

Multimodal response generation in GIS

Adaptive multimodal recognition of voluntary and involuntary gestures of people with motor disabilities

Using language structure for adaptive multimodal language acquisition

Content Provider	ACM Digital Library
Author	Chandrasekaran, Rajesh
Abstract	In human spoken communication, language structure plays a vital role in providing a framework for humans to understand each other. Using language rules, words are combined into meaningful sentences to represent knowledge. Speech enabled systems based on pre-programmed Rule Grammar suffer from constraints on vocabulary and sentence structures. To address this problem, in this paper, we discuss a language acquisition system that is capable of learning new words and their corresponding semantic meaning by initiating an adaptive dialog with the user. Thus, the vocabulary of the system can be increased in real time by the user. The language acquisition system is provided knowledge about language structure and is capable of accepting multimodal user inputs that includes speech, touch, pen-tablet, mouse, and keyboard. We discuss the efficiency of learning new concepts and the ease with which users can teach the system new concepts. The multimodal language acquisition system is capable of acquiring, in real time, new words that pertain to objects, actions or attributes and their corresponding meanings. The first step in this process is to detect unknown words in the spoken utterance. Any new word that is detected is classified into one of the above mentioned categories. The second step is to learn from the user the meaning of the word and add it to the semantic database. An unknown word is flagged whenever an utterance is not consistent with the pre-programmed Rule Grammar. Because the system can acquire words pertaining to objects, actions or attributes, we are interested in words that are nouns, verbs or adjectives. We use a transformation based part-of-speech tagger that is capable of annotating English words with their part-of-speech to identify words in the utterance that are nouns, verbs and adjectives. These words are searched in the semantic database and unknown words are identified. The system then initiates an adaptive dialog with the user, requesting the user to provide the meaning of the unknown word. When the user has provided the relevant meaning using any of the input modalities, the system checks whether the meaning given corresponds to the category of the word, i.e. if the unknown word is a noun then the user can associate only an object with it or if the unknown word is a verb then only an action can be associated with the word. Thus, the system uses the knowledge of the occurrence of the word in the sentence to determine what kind of meaning can be associated with the word. The language structure thus gives the system a basic knowledge of the unknown word.
Starting Page	345
Ending Page	345
Page Count	1
File Format	PDF
ISBN	1581139950
DOI	10.1145/1027933.1028003
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2004-10-13
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Computer language learning Language structure Adaptive dialogue systems Language acquisition
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in