NDLI: Maximum Likelihood Estimation of information measures

Content Provider	IEEE Xplore Digital Library
Author	Jiantao Jiao Venkat, K. Yanjun Han Weissman, T.
Copyright Year	2015
Description	Author affiliation: Stanford Univ., Stanford, CA, USA (Jiantao Jiao; Venkat, K.; Weissman, T.) \|\| Tsinghua Univ., Beijing, China (Yanjun Han)
Abstract	The Maximum Likelihood Estimator (MLE) is widely used in estimating information measures, and involves “plugging-in” the empirical distribution of the data to estimate a given functional of the unknown distribution. In this work we propose a general framework and procedure to analyze the nonasymptotic performance of the MLE in estimating functionals of discrete distributions, under the worst-case mean squared error criterion. We show that existing theory is insufficient for analyzing the bias of the MLE, and propose to apply the theory of approximation using positive linear operators to study this bias. The variance is controlled using the well-known tools from the literature on concentration inequalities. Our techniques completely characterize the maximum $L_{2}$ risk incurred by the MLE in estimating the Shannon entropy H(P) = $Σ_{i=1}^{S}$ $-p_{i}ln$ $p_{i},$ and $F_{α}(P)$ = $Σ_{i=1}^{S}p_{i}^{α}$ up to a multiplicative constant. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have n ≪ S observations for the MLE to be consistent, where S represents the support size. In addition, we obtain that it is necessary and sufficient to consider n ≪ $S^{1/α}$ samples for the MLE to consistently estimate $F_{α}(P);$ 0 <;α <; 1. The minimax rate-optimal estimators for both problems require S/ln S and $S^{1/α}$ / ln S samples, which implies that the MLE is strictly sub-optimal. When 1 <; α <; 3/2, we show that the maximum $L_{2}$ rate of convergence for the MLE is $n^{-2(α-1)}$ for infinite support size, while the minimax $L_{2}$ rate is (n ln $n)^{-2(α-1)}.$ When α ≥ 3/2, the MLE achieves the minimax optimal $L_{2}$ convergence rate $n^{-1}$ regardless of the support size.
Starting Page	839
Ending Page	843
File Size	290456
Page Count	5
File Format	PDF
ISSN	21578117
e-ISBN	9781467377041
DOI	10.1109/ISIT.2015.7282573
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2015-06-14
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Maximum likelihood estimation Entropy Approximation methods Polynomials Convergence Information theory
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Minimax estimation of information measures

Penalized maximum likelihood methods for finite memory estimators of infinite memory processes

Minimax Estimation of Functionals of Discrete Distributions

Does dirichlet prior smoothing solve the Shannon entropy estimation problem?

Optimisation of the Maximum Likelihood Method Using Bias Minimisation

Maximum Likelihood DOA Estimation Based on the Cross-Entropy Method

Model-fitting in the presence of outliers

SCHEME: Stochastically convergent heuristical Expectation Maximization estimation

Sketching and Streaming Entropy via Approximation Theory

Maximum Likelihood Estimation of information measures

Similar Documents

Minimax estimation of information measures

Penalized maximum likelihood methods for finite memory estimators of infinite memory processes

Minimax Estimation of Functionals of Discrete Distributions

Does dirichlet prior smoothing solve the Shannon entropy estimation problem?

Optimisation of the Maximum Likelihood Method Using Bias Minimisation

Maximum Likelihood DOA Estimation Based on the Cross-Entropy Method

Model-fitting in the presence of outliers

SCHEME: Stochastically convergent heuristical Expectation Maximization estimation

Sketching and Streaming Entropy via Approximation Theory

Maximum Likelihood Estimation of information measures