NDLI: An experimental study of the intrinsic stability of random forest variable importance measures

Content Provider	Springer Nature : BioMed Central
Author	Wang, Huazhen Yang, Fan Luo, Zhiyuan
Abstract	Background The stability of Variable Importance Measures (VIMs) based on random forest has recently received increased attention. Despite the extensive attention on traditional stability of data perturbations or parameter variations, few studies include influences coming from the intrinsic randomness in generating VIMs, i.e. bagging, randomization and permutation. To address these influences, in this paper we introduce a new concept of intrinsic stability of VIMs, which is defined as the self-consistence among feature rankings in repeated runs of VIMs without data perturbations and parameter variations. Two widely used VIMs, i.e., Mean Decrease Accuracy (MDA) and Mean Decrease Gini (MDG) are comprehensively investigated. The motivation of this study is two-fold. First, we empirically verify the prevalence of intrinsic stability of VIMs over many real-world datasets to highlight that the instability of VIMs does not originate exclusively from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. Second, through Spearman and Pearson tests we comprehensively investigate how different factors influence the intrinsic stability. Results The experiments are carried out on 19 benchmark datasets with diverse characteristics, including 10 high-dimensional and small-sample gene expression datasets. Experimental results demonstrate the prevalence of intrinsic stability of VIMs. Spearman and Pearson tests on the correlations between intrinsic stability and different factors show that #feature (number of features) and #sample (size of sample) have a coupling effect on the intrinsic stability. The synthetic indictor, #feature/#sample, shows both negative monotonic correlation and negative linear correlation with the intrinsic stability, while OOB accuracy has monotonic correlations with intrinsic stability. This indicates that high-dimensional, small-sample and high complexity datasets may suffer more from intrinsic instability of VIMs. Furthermore, with respect to parameter settings of random forest, a large number of trees is preferred. No significant correlations can be seen between intrinsic stability and other factors. Finally, the magnitude of intrinsic stability is always smaller than that of traditional stability. Conclusion First, the prevalence of intrinsic stability of VIMs demonstrates that the instability of VIMs not only comes from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. This finding gives a better understanding of VIM stability, and may help reduce the instability of VIMs. Second, by investigating the potential factors of intrinsic stability, users would be more aware of the risks and hence more careful when using VIMs, especially on high-dimensional, small-sample and high complexity datasets.
Related Links	https://bmcbioinformatics.biomedcentral.com/counter/pdf/10.1186/s12859-016-0900-5.pdf
Ending Page	18
Page Count	18
Starting Page	1
File Format	HTM / HTML
ISSN	14712105
DOI	10.1186/s12859-016-0900-5
Journal	BMC Bioinformatics
Issue Number	1
Volume Number	17
Language	English
Publisher	BioMed Central
Publisher Date	2016-02-03
Access Restriction	Open
Subject Keyword	Bioinformatics Microarrays Computational Biology Computer Appl. in Life Sciences Algorithms Random forest Variable importance measure Stability Feature selection Computational Biology/Bioinformatics
Content Type	Text
Resource Type	Article
Subject	Molecular Biology Biochemistry Computer Science Applications Applied Mathematics Structural Biology
Journal Impact Factor	2.9/2023
5-Year Journal Impact Factor	3.6/2023

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Bias in random forest variable importance measures: Illustrations, sources and a solution

Intervention in prediction measure: a new approach to assessing variable importance for random forests

Robustness of Random Forest-based gene selection methods

An AUC-based permutation variable importance measure for random forests

Conditional variable importance for random forests

Gene selection and classification of microarray data using random forest

A random forest based computational model for predicting novel lncRNA-disease associations

On consensus biomarker selection

The behaviour of random forest permutation-based variable importance measures under predictor correlation

An experimental study of the intrinsic stability of random forest variable importance measures

Similar Documents

Bias in random forest variable importance measures: Illustrations, sources and a solution

Intervention in prediction measure: a new approach to assessing variable importance for random forests

Robustness of Random Forest-based gene selection methods

An AUC-based permutation variable importance measure for random forests

Conditional variable importance for random forests

Gene selection and classification of microarray data using random forest

A random forest based computational model for predicting novel lncRNA-disease associations

On consensus biomarker selection

The behaviour of random forest permutation-based variable importance measures under predictor correlation

An experimental study of the intrinsic stability of random forest variable importance measures