NDLI: Noise benefits in backpropagation and deep bidirectional pre-training

Content Provider	IEEE Xplore Digital Library
Author	Audhkhasi, K. Osoba, O. Kosko, B.
Copyright Year	2013
Description	Author affiliation: Electr. Eng. Dept., Univ. of Southern California, Los Angeles, CA, USA (Audhkhasi, K.; Osoba, O.; Kosko, B.)
Abstract	We prove that noise can speed convergence in the backpropagation algorithm. The proof consists of two separate results. The first result proves that the backpropagation algorithm is a special case of the generalized Expectation-Maximization (EM) algorithm for iterative maximum likelihood estimation. The second result uses the recent EM noise benefit to derive a sufficient condition for backpropagation training. The noise adds directly to the training data. A noise benefit also applies to the deep bidirectional pre-training of the neural network as well as to the backpropagation training of the network. The geometry of the noise benefit depends on the probability structure of the neurons at each layer. Logistic sigmoidal neurons produce a forbidden noise region that lies below a hyperplane. Then all noise on or above the hyperplane can only speed convergence of the neural network. The forbidden noise region is a sphere if the neurons have a Gaussian signal or activation function. These noise benefits all follow from the general noise benefit of the EM algorithm. Monte Carlo sample means estimate the population expectations in the EM algorithm. We demonstrate the noise benefits using MNIST digit classification.
Starting Page	1
Ending Page	8
File Size	434038
Page Count	8
File Format	PDF
ISSN	21614407
e-ISBN	9781467361293
DOI	10.1109/IJCNN.2013.6707022
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2013-08-04
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Noise Neurons Biological neural networks Backpropagation Training Convergence Maximum likelihood estimation Expectation-Maximization algorithm bidirectional associative memory noise benefit neural network stochastic resonance
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Noise benefits in the expectation-maximization algorithm: Nem theorems and models

tutorial on Expectation-Maximization algorithm including Maximum-Likelihood Estimation and EM training of (2003)

Encoding strategy for maximum noise tolerance bidirectional associative memory.

Learning Associative Memories by Error Backpropagation

Sensitivity to noise in bidirectional associative memory (BAM)

Neural pre-coding increases the pattern retrieval capacity of Hopfield and Bidirectional Associative Memories

Fast Backpropagation Neural Network algorithm for reducing convergence time of BPNN image compression

Noisy hidden Markov models for speech recognition

Delay-Dependent Robust Exponential Stability Analysis of Stochastic Bidirectional Associative Memory Neural Networks with Delays

Noise benefits in backpropagation and deep bidirectional pre-training

Similar Documents

Noise benefits in the expectation-maximization algorithm: Nem theorems and models

tutorial on Expectation-Maximization algorithm including Maximum-Likelihood Estimation and EM training of (2003)

Encoding strategy for maximum noise tolerance bidirectional associative memory.

Learning Associative Memories by Error Backpropagation

Sensitivity to noise in bidirectional associative memory (BAM)

Neural pre-coding increases the pattern retrieval capacity of Hopfield and Bidirectional Associative Memories

Fast Backpropagation Neural Network algorithm for reducing convergence time of BPNN image compression

Noisy hidden Markov models for speech recognition

Delay-Dependent Robust Exponential Stability Analysis of Stochastic Bidirectional Associative Memory Neural Networks with Delays

Noise benefits in backpropagation and deep bidirectional pre-training