NDLI: Parametrized Actor-Critic Algorithms for Finite-Horizon MDPs

Content Provider	IEEE Xplore Digital Library
Author	Abdulla, M.S. Bhatnagar, S.
Copyright Year	2007
Description	Author affiliation: Indian Inst. of Sci., Bangalore (Abdulla, M.S.; Bhatnagar, S.)
Abstract	Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FH-MDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.
Starting Page	534
Ending Page	539
File Size	280005
Page Count	6
File Format	PDF
ISBN	1424409888
ISSN	07431619
DOI	10.1109/ACC.2007.4282587
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2007-07-09
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Stochastic processes Computational modeling Approximation algorithms Table lookup Costs Convergence Communication system control Cities and towns Computer science Automation actor-critic algorithms Finite horizon Markov decision processes reinforcement learning two timescale stochastic approximation
Content Type	Text
Resource Type	Article
Subject	Electrical and Electronic Engineering

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A Reinforcement Learning Based Algorithm for Finite Horizon Markov Decision Processes

Parametrized actor-critic algorithms for finite-horizon mdps (2007).

Solving mdps using two-timescale simulated annealing with multiplicative weights.

Parametrized Actor-critic Algorithms For Finite-horizon MDPs

Simulation Based Algorithms For Markov Decision Process And Stochastic Optimization

Actor-Critic--Type Learning Algorithms for Markov Decision Processes

Solving MDPs using Two-timescale Simulated Annealing with Multiplicative Weights

Natural Actor–Critic Algorithms (2009)

OnActor-Critic Algorithms

Parametrized Actor-Critic Algorithms for Finite-Horizon MDPs

Similar Documents

A Reinforcement Learning Based Algorithm for Finite Horizon Markov Decision Processes

Parametrized actor-critic algorithms for finite-horizon mdps (2007).

Solving mdps using two-timescale simulated annealing with multiplicative weights.

Parametrized Actor-critic Algorithms For Finite-horizon MDPs

Simulation Based Algorithms For Markov Decision Process And Stochastic Optimization

Actor-Critic--Type Learning Algorithms for Markov Decision Processes

Solving MDPs using Two-timescale Simulated Annealing with Multiplicative Weights

Natural Actor–Critic Algorithms (2009)

OnActor-Critic Algorithms

Parametrized Actor-Critic Algorithms for Finite-Horizon MDPs