NDLI: Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

Please wait, while we are loading the content...

Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

Content Provider	ACM Digital Library
Author	Young, Steve Jurčíček, Filip Thomson, Blaise
Copyright Year	2011
Abstract	This article presents a novel algorithm for learning parameters in statistical dialogue systems which are modeled as Partially Observable Markov Decision Processes (POMDPs). The three main components of a POMDP dialogue manager are a dialogue model representing dialogue state information; a policy that selects the system's responses based on the inferred state; and a reward function that specifies the desired behavior of the system. Ideally both the model parameters and the policy would be designed to maximize the cumulative reward. However, while there are many techniques available for learning the optimal policy, no good ways of learning the optimal model parameters that scale to real-world dialogue systems have been found yet. The presented algorithm, called the Natural Actor and Belief Critic (NABC), is a policy gradient method that offers a solution to this problem. Based on observed rewards, the algorithm estimates the natural gradient of the expected cumulative reward. The resulting gradient is then used to adapt both the prior distribution of the dialogue model parameters and the policy parameters. In addition, the article presents a variant of the NABC algorithm, called the Natural Belief Critic (NBC), which assumes that the policy is fixed and only the model parameters need to be estimated. The algorithms are evaluated on a spoken dialogue system in the tourist information domain. The experiments show that model parameters estimated to maximize the expected cumulative reward result in significantly improved performance compared to the baseline hand-crafted model parameters. The algorithms are also compared to optimization techniques using plain gradients and state-of-the-art random search algorithms. In all cases, the algorithms based on the natural gradient work significantly better.
Starting Page	1
Ending Page	26
Page Count	26
File Format	PDF
ISSN	15504875
e-ISSN	15504883
DOI	10.1145/1966407.1966411
Volume Number	7
Issue Number	3
Journal	ACM Transactions on Speech and Language Processing (TSLP)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2011-06-06
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	POMDP Spoken dialogue systems Reinforcement learning
Content Type	Text
Resource Type	Article
Subject	1700/1701 Computational Mathematics

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Effective handling of dialogue state in the hidden information state POMDP-based dialogue manager

2010. Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems

Sample-efficient batch reinforcement learning for dialogue management optimization

Bayesian reinforcement learning for POMDP-based dialogue systems

Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems (2010)

Bayesian reinforcement learning for POMDP-based dialogue systems (2011)

Modeling spoken decision support dialogue and optimization of its dialogue strategy

Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems (2010)

ACIS: An Improved Actor-Critic Method for POMDPs with Internal State

Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs

Similar Documents

Effective handling of dialogue state in the hidden information state POMDP-based dialogue manager

2010. Bayesian update of dialogue state: A pomdp framework for spoken dialogue systems

Sample-efficient batch reinforcement learning for dialogue management optimization

Bayesian reinforcement learning for POMDP-based dialogue systems

Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems (2010)

Bayesian reinforcement learning for POMDP-based dialogue systems (2011)

Modeling spoken decision support dialogue and optimization of its dialogue strategy

Natural Belief-Critic: a reinforcement algorithm for parameter estimation in statistical spoken dialogue systems (2010)

ACIS: An Improved Actor-Critic Method for POMDPs with Internal State

Natural actor and belief critic: Reinforcement algorithm for learning parameters of dialogue systems modelled as POMDPs