NDLI: An analysis of noise in recurrent neural networks: convergence and generalization

Content Provider	IEEE Xplore Digital Library
Author	Kam-Chuen Jim Giles, C.L. Horne, B.G.
Copyright Year	1990
Abstract	Concerns the effect of noise on the performance of feedforward neural nets. We introduce and analyze various methods of injecting synaptic noise into dynamically driven recurrent nets during training. Theoretical results show that applying a controlled amount of noise during training may improve convergence and generalization performance. We analyze the effects of various noise parameters and predict that best overall performance can be achieved by injecting additive noise at each time step. Noise contributes a second-order gradient term to the error function which can be viewed as an anticipatory agent to aid convergence. This term appears to find promising regions of weight space in the beginning stages of training when the training error is large and should improve convergence on error surfaces with local minima. The first-order term is a regularization term that can improve generalization. Specifically, it can encourage internal representations where the state nodes operate in the saturated regions of the sigmoid discriminant function. While this effect can improve performance on automata inference problems with binary inputs and target outputs, it is unclear what effect it will have on other types of problems. To substantiate these predictions, we present simulations on learning the dual parity grammar from temporal strings for all noise models, and present simulations on learning a randomly generated six-state grammar using the predicted best noise model.
Sponsorship	IEEE Computational Intelligence Society
Starting Page	1424
Ending Page	1438
Page Count	15
File Size	1466329
File Format	PDF
ISSN	10459227
Volume Number	7
Issue Number	6
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	1996-11-01
Publisher Place	U.S.A.
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Intelligent networks Neural networks Convergence Predictive models Additive noise Temperature Feedforward neural networks Noise generators Central Processing Unit State-space methods
Content Type	Text
Resource Type	Article
Subject	Artificial Intelligence Computer Networks and Communications Computer Science Applications Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Using additive noise in back-propagation training

Markovian architectural bias of recurrent neural networks

Neural network applications for jamming state information generator

Recurrent neural networks and robust time series prediction

Diagonal recurrent neural networks for dynamic systems control

Backpropagation of pseudo-errors: neural networks that are adaptive to heterogeneous noise

A recurrent Newton algorithm and its convergence properties

Pruning recurrent neural networks for improved generalization performance

Financial volatility trading using recurrent neural networks

An analysis of noise in recurrent neural networks: convergence and generalization

Similar Documents

Using additive noise in back-propagation training

Markovian architectural bias of recurrent neural networks

Neural network applications for jamming state information generator

Recurrent neural networks and robust time series prediction

Diagonal recurrent neural networks for dynamic systems control

Backpropagation of pseudo-errors: neural networks that are adaptive to heterogeneous noise

A recurrent Newton algorithm and its convergence properties

Pruning recurrent neural networks for improved generalization performance

Financial volatility trading using recurrent neural networks

An analysis of noise in recurrent neural networks: convergence and generalization