NDLI: Online Reinforcement Learning for Dynamic Multimedia Systems

Content Provider	IEEE Xplore Digital Library
Author	Mastronarde, N. van der Schaar, M.
Copyright Year	1992
Abstract	In our previous work, we proposed a systematic cross-layer framework for dynamic multimedia systems, which allows each layer to make autonomous and foresighted decisions that maximize the system's long-term performance, while meeting the application's real-time delay constraints. The proposed solution solved the cross-layer optimization offline, under the assumption that the multimedia system's probabilistic dynamics were known a priori, by modeling the system as a layered Markov decision process. In practice, however, these dynamics are unknown a priori and, therefore, must be learned online. In this paper, we address this problem by allowing the multimedia system layers to learn, through repeated interactions with each other, to autonomously optimize the system's long-term performance at run-time. The two key challenges in this layered learning setting are: (i) each layer's learning performance is directly impacted by not only its own dynamics, but also by the learning processes of the other layers with which it interacts; and (ii) selecting a learning model that appropriately balances time-complexity (i.e., learning speed) with the multimedia system's limited memory and the multimedia application's real-time delay constraints. We propose two reinforcement learning algorithms for optimizing the system under different design constraints: the first algorithm solves the cross-layer optimization in a centralized manner and the second solves it in a decentralized manner. We analyze both algorithms in terms of their required computation, memory, and interlayer communication overheads. After noting that the proposed reinforcement learning algorithms learn too slowly, we introduce a complementary accelerated learning algorithm that exploits partial knowledge about the system's dynamics in order to dramatically improve the system's performance. In our experiments, we demonstrate that decentralized learning can perform equally as well as centralized learning, while enabling the layers to act autonomously. Additionally, we show that existing application-independent reinforcement learning algorithms, and existing myopic learning algorithms deployed in multimedia systems, perform significantly worse than our proposed application-aware and foresighted learning methods.
Sponsorship	IEEE Signal Processing Society
Page Count	16
File Size	1663597
Starting Page	290
Ending Page	305
File Format	PDF
ISSN	10577149
Volume Number	19
Issue Number	2
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2010-02-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Learning Multimedia systems Real time systems Delay Constraint optimization Design optimization Algorithm design and analysis Runtime Acceleration System performance resource-constrained multimedia system Cross-layer multimedia system optimization layered Markov decision process reinforcement learning
Content Type	Text
Resource Type	Article
Subject	Computer Graphics and Computer-Aided Design Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Towards a General Framework for Cross-Layer Decision Making in Multimedia Systems

Sample Efficient Constrained Reinforcement Learning

Constrained Markov control model and online stochastic optimization algorithm for power conservation in multimedia server cluster systems

${Q}$-Learning Algorithms for Constrained Markov Decision Processes With Randomized Monotone Policies: Application to MIMO Transmission Control

Decomposition Principles and Online Learning in Cross-Layer Optimization for Delay-Sensitive Applications

Constrained optimization based on hybrid evolutionary algorithm and adaptive constraint-handling technique

On-Line Learning and Optimization for Wireless Video Transmission

A UoI-Optimal Policy for Timely Status Updates with Resource Constraint

Symbolic debugging of embedded hardware and software

Online Reinforcement Learning for Dynamic Multimedia Systems

Similar Documents

Towards a General Framework for Cross-Layer Decision Making in Multimedia Systems

Sample Efficient Constrained Reinforcement Learning

Constrained Markov control model and online stochastic optimization algorithm for power conservation in multimedia server cluster systems

${Q}$-Learning Algorithms for Constrained Markov Decision Processes With Randomized Monotone Policies: Application to MIMO Transmission Control

Decomposition Principles and Online Learning in Cross-Layer Optimization for Delay-Sensitive Applications

Constrained optimization based on hybrid evolutionary algorithm and adaptive constraint-handling technique

On-Line Learning and Optimization for Wireless Video Transmission

A UoI-Optimal Policy for Timely Status Updates with Resource Constraint

Symbolic debugging of embedded hardware and software

Online Reinforcement Learning for Dynamic Multimedia Systems