L G ] 2 8 M ay 2 01 9 Gradient Descent Finds Global Minima of Deep Neural Networks

Content Provider	Semantic Scholar
Author	Du, Simon S. Lee, Jason D. Li, Haochuan Wang, Liwei Zhai, Xiyu
Copyright Year	2019
Abstract	Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep overparameterized neural network with residual connections (ResNet). Our analysis relies on the particular structure of the Gram matrix induced by the neural network architecture. This structure allows us to show the Gram matrix is stable throughout the training process and this stability implies the global optimality of the gradient descent algorithm. We further extend our analysis to deep residual convolutional neural networks and obtain a similar convergence result.
File Format	PDF HTM / HTML
Alternate Webpage(s)	http://arxiv-export-lb.library.cornell.edu/pdf/1811.03804
Language	English
Access Restriction	Open
Content Type	Text
Resource Type	Article