NDLI: The second order temporal difference error for Sarsa(λ)

Please wait, while we are loading the content...

2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)

2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)

Local stability analysis of high-order recurrent neural networks with multi-step piecewise linear activation functions

Finite-horizon optimal control design for uncertain linear discrete-time systems

Adaptive optimal control for nonlinear discrete-time systems

Optimal control for a class of nonlinear system with controller constraints based on finite-approximation-errors ADP algorithm

Finite horizon stochastic optimal control of uncertain linear networked control system

Exponential moving average Q-learning algorithm

Real-time tracking on adaptive critic design with uniformly ultimately bounded condition

A novel approach for constructing basis functions in approximate dynamic programming for feedback control

A combined hierarchical reinforcement learning based approach for multi-robot cooperative target searching in complex unknown environments

The second order temporal difference error for Sarsa(λ)

Optimistic planning for continuous-action deterministic systems

Optimistic planning for belief-augmented Markov Decision Processes

Optimized look-ahead trees: Extensions to large and continuous action spaces

Bias-corrected Q-learning to control max-operator bias in Q-learning

Value function approximation and model predictive control

Reinforcement learning in the game of Othello: Learning against a fixed opponent and learning from self-play

A reinforcement learning algorithm developed to model GenCo strategic bidding behavior in multidimensional and continuous state and action spaces

Free energy based policy gradients

Delayed insertion and rule effect moderation of domain knowledge for reinforcement learning

Exploring the relationship of reward and punishment in reinforcement learning

Analyzing collective behavior in evolutionary swarm robotic systems based on an ethological approach

Reinforcement learning to train Ms. Pac-Man using higher-order action-relative inputs

A study on the efficiency of learning a robot controller in various environments

On the coordination system for the dimensionality-reduced inputs of mario

Optimal control for a class of nonlinear systems with state delay based on Adaptive Dynamic Programming with ε-error bound

An integrated design for intensified direct heuristic dynamic programming

Scalarized multi-objective reinforcement learning: Novel design techniques

Fault accommodation for complete synchronization of complex neural networks

2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)

2009 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning

The second order temporal difference error for Sarsa(λ)

Content Provider	IEEE Xplore Digital Library
Author	Qiming Fu Quan Liu Fei Xiao Guixin Chen
Copyright Year	2013
Description	Author affiliation: Dept. of Comput. Sci. & Technol., Soochow Univ., Suzhou, China (Qiming Fu; Quan Liu; Fei Xiao; Guixin Chen)
Abstract	Traditional reinforcement learning algorithms, such as Q-learning, Q(λ), Sarsa, and Sarsa(λ), update the action value function using temporal difference (TD) error, which is computed by the last action value function. From the perspective of the TD error, and with respect to the problems of low efficiency and slow convergence of the traditional Sarsa(λ) algorithm, this paper defines the $n^{th}$ order TD Error, applies it in the traditional Sarsa(λ) algorithm, and develops a fast Sarsa(λ) algorithm based on the $2^{nd}$ order TD Error. The algorithm adjusts the Q value with the second-order TD Error and broadcasts the TD Error into the whole state-action space, which speeds up the convergence of the algorithm. This paper also analyzes the convergence rate, and under the condition of one-step update, the results show that the number of iteration depends primarily on γ, ε. Finally, using the proposed algorithm on the traditional reinforcement learning problems, the results show that the algorithm has both a faster convergence rate and better convergence performance.
Starting Page	60
Ending Page	68
File Size	463636
Page Count	9
File Format	PDF
ISBN	9781467359252
ISSN	23251824
DOI	10.1109/ADPRL.2013.6614990
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-04-16
Publisher Place	Singapore
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Algorithm design and analysis Machine learning algorithms Eligibility Trace Sarsa(λ) Algorithm Learning (artificial intelligence) Reinforcement Learning Second Order TD Error Educational institutions Mathematical model Markov Decision Process Equations Convergence
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Reinforcement learning with replacing eligibility traces

Fast and Efficient Reinforcement Learning with Truncated Temporal Differences (1995)

Multi-criteria expertness based cooperative method for SARSA and eligibility trace algorithms

Multi-robot collaboration based on Markov decision process in Robocup3D soccer simulation game

Efficient asymptotic approximation in temporal difference learning.

The convergence of TD(λ) for general λ

The Convergence of TD(λ) for General λ

TD(λ) Networks: Temporal-Difference Networks with Eligibility Traces

The Analysis of Experimental Results of Reinforcement Learning Systems

The second order temporal difference error for Sarsa(λ)

Similar Documents

Reinforcement learning with replacing eligibility traces

Fast and Efficient Reinforcement Learning with Truncated Temporal Differences (1995)

Multi-criteria expertness based cooperative method for SARSA and eligibility trace algorithms

Multi-robot collaboration based on Markov decision process in Robocup3D soccer simulation game

Efficient asymptotic approximation in temporal difference learning.

The convergence of TD(λ) for general λ

The Convergence of TD(λ) for General λ

TD(λ) Networks: Temporal-Difference Networks with Eligibility Traces

The Analysis of Experimental Results of Reinforcement Learning Systems

The second order temporal difference error for Sarsa(λ)