NDLI: An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Content Provider	Springer Nature Link
Author	Bauer, Eric Kohavi, Ron
Copyright Year	1999
Abstract	Methods for voting classification algorithms, such as Bagging and AdaBoost, have been shown to be very successful in improving the accuracy of certain classifiers for artificial and real-world datasets. We review these algorithms and describe a large empirical study comparing several variants in conjunction with a decision tree inducer (three variants) and a Naive-Bayes inducer. The purpose of the study is to improve our understanding of why and when these algorithms, which use perturbation, reweighting, and combination techniques, affect classification error. We provide a bias and variance decomposition of the error to show how different methods and variants influence these two terms. This allowed us to determine that Bagging reduced variance of unstable methods, while boosting methods (AdaBoost and Arc-x4) reduced both the bias and variance of unstable methods but increased the variance for Naive-Bayes, which was very stable. We observed that Arc-x4 behaves differently than AdaBoost if reweighting is used instead of resampling, indicating a fundamental difference. Voting variants, some of which are introduced in this paper, include: pruning versus no pruning, use of probabilistic estimates, weight perturbations (Wagging), and backfitting of data. We found that Bagging improves when probabilistic estimates in conjunction with no-pruning are used, as well as when the data was backfit. We measure tree sizes and show an interesting positive correlation between the increase in the average tree size in AdaBoost trials and its success in reducing the error. We compare the mean-squared error of voting methods to non-voting methods and show that the voting methods lead to large and significant reductions in the mean-squared errors. Practical problems that arise in implementing boosting algorithms are explored, including numerical instabilities and underflows. We use scatterplots that graphically show how AdaBoost reweights instances, emphasizing not only “hard” areas but also outliers and noise.
Starting Page	105
Ending Page	139
Page Count	35
File Format	PDF
ISSN	08856125
Journal	Machine Learning
Volume Number	36
Issue Number	1-2
e-ISSN	15730565
Language	English
Publisher	Kluwer Academic Publishers
Publisher Date	1999-01-01
Publisher Place	Boston
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Computing Methodologies Artificial Intelligence (incl. Robotics) Simulation and Modeling Language Translation and Linguistics Automation and Robotics
Content Type	Text
Resource Type	Article
Subject	Artificial Intelligence Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Bagging Equalizes Influence

Boosting and Microarray Data

Density estimation with stagewise optimization of the empirical risk

Multi-Class Learning by Smoothed Boosting

Extending boosting for large scale spoken language understanding

Naive Bayesian Classification of Structured Data

Functional Trees

Correlation Clustering

Random Forests

An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants

Similar Documents

Bagging Equalizes Influence

Boosting and Microarray Data

Density estimation with stagewise optimization of the empirical risk

Multi-Class Learning by Smoothed Boosting

Extending boosting for large scale spoken language understanding

Naive Bayesian Classification of Structured Data

Functional Trees

Correlation Clustering

Random Forests

An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants