NDLI: Primal-dual algorithms for discounted Markov decision processes

Content Provider	IEEE Xplore Digital Library
Author	Cogill, R.
Copyright Year	2015
Description	Author affiliation: IBM Res. Ireland, Dublin, Ireland (Cogill, R.)
Abstract	Several well-known algorithms in the field of combinatorial optimization can be interpreted in terms of the primal-dual method for solving linear programs. For example, Dijkstra's algorithm, the Ford-Fulkerson algorithm, and the Hungarian algorithm can all be viewed as the primal-dual method applied to the linear programming formulations of their respective optimization problems. Roughly speaking, successfully applying the primal-dual method to an optimization problem that can be posed as a linear program relies on the ability to find a simple characterization of the optimal solutions to a related linear program, called the `dual of the restricted primal' (DRP). This paper is motivated by the following question: What is the algorithm we obtain if we apply the primal-dual method to a linear programming formulation of a discounted cost Markov decision process? We will first show that a widely-used variant of the value iteration algorithm for Markov decision processes can be interpreted in terms of the primal-dual method, where the value function is updated with suboptimal solutions to the DRP in each iteration. We then provide the optimal solution to the DRP in closed-form, and present the algorithm that results when using this solution to update the value function in each iteration. Unlike the algorithms obtained from suboptimal DRP updates, this algorithm is guaranteed to yield the optimal value function in a finite number of iterations. Finally, we show that the iterations of the primal-dual algorithm can be interpreted as repeated application of the policy iteration algorithm to a special class of Markov decision processes. When considered alongside recent results characterizing the computational complexity of the policy iteration algorithm, this observation could provide new insights into the computational complexity of solving discounted-cost Markov decision processes.
Starting Page	260
Ending Page	265
File Size	160476
Page Count	6
File Format	PDF
e-ISBN	9783952426937
DOI	10.1109/ECC.2015.7330554
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-07-15
Publisher Place	Austria
Access Restriction	Subscribed
Rights Holder	EUCA
Subject Keyword	Computational modeling Markov processes Linear programming Polynomials Computational complexity Optimization Convergence
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

An Analysis of Primal-Dual Algorithms for Discounted Markov Decision Processes

The complexity of Policy Iteration is exponential for discounted Markov Decision Processes

Toward an optimized value iteration algorithm for average cost Markov decision processes

Linear computational complexity design of constrained optimal ILC

The distance measure on consensus decision making based on primal-dual programming models

Theoretical convergence of large-step primal-dual interior point algorithms for linear programming (1991).

An empirical algorithm for relative value iteration for average-cost MDPs

Theoretical Convergence of Large-Step Primal-Dual Interior Point Algorithms for Linear Programming (1992)

Policy Gradient Semi-markov Decision Process

Primal-dual algorithms for discounted Markov decision processes

Similar Documents

An Analysis of Primal-Dual Algorithms for Discounted Markov Decision Processes

The complexity of Policy Iteration is exponential for discounted Markov Decision Processes

Toward an optimized value iteration algorithm for average cost Markov decision processes

Linear computational complexity design of constrained optimal ILC

The distance measure on consensus decision making based on primal-dual programming models

Theoretical convergence of large-step primal-dual interior point algorithms for linear programming (1991).

An empirical algorithm for relative value iteration for average-cost MDPs

Theoretical Convergence of Large-Step Primal-Dual Interior Point Algorithms for Linear Programming (1992)

Policy Gradient Semi-markov Decision Process

Primal-dual algorithms for discounted Markov decision processes