NDLI: An Efficient Corpus-Based Stemmer

Please wait, while we are loading the content...

An Efficient Corpus-Based Stemmer

Content Provider	Springer Nature Link
Author	Singh, Jasmeet Gupta, Vishal
Copyright Year	2017
Abstract	Word stemming is a linguistic process in which the various inflected word forms are matched to their base form. It is among the basic text pre-processing approaches used in Natural Language Processing and Information Retrieval. Stemming is employed at the text pre-processing stage to solve the issue of vocabulary mismatch or to reduce the size of the word vocabulary, and consequently also the dimensionality of training data for statistical models. In this article, we present a fully unsupervised corpus-based text stemming method which clusters morphologically related words based on lexical knowledge. The proposed method performs cognitive-inspired computing to discover morphologically related words from the corpus without any human intervention or language-specific knowledge. The performance of the proposed method is evaluated in inflection removal (approximating lemmas) and Information Retrieval tasks. The retrieval experiments in four different languages using standard Text Retrieval Conference, Cross-Language Evaluation Forum, and Forum for Information Retrieval Evaluation collections show that the proposed stemming method performs significantly better than no stemming. In the case of highly inflectional languages, Marathi and Hungarian, the improvement in Mean Average Precision is nearly 50% as compared to unstemmed words. Moreover, the proposed unsupervised stemming method outperforms state-of-the-art strong language-independent and rule-based stemming methods in all the languages. Besides Information Retrieval, the proposed stemming method also performs significantly better in inflection removal experiments. The proposed unsupervised language-independent stemming method can be used as a multipurpose tool for various tasks such as the approximation of lemmas, improving retrieval performance or other Natural Language Processing applications.
Starting Page	671
Ending Page	688
Page Count	18
File Format	PDF
ISSN	18669956
Journal	Cognitive Computation
Volume Number	9
Issue Number	5
e-ISSN	18669964
Language	English
Publisher	Springer US
Publisher Date	2017-06-07
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Stemming Information retrieval Natural language processing Neurosciences Computation by Abstract Devices Artificial Intelligence (incl. Robotics) Computational Biology/Bioinformatics
Content Type	Text
Resource Type	Article
Subject	Cognitive Neuroscience Computer Science Applications Computer Vision and Pattern Recognition

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Cognitive Computation: An Introduction

Computational Creativity, Intelligence and Autonomy

Sentic Computing

BLISS: an Artificial Language for Learnability Studies

Special Issue Editorial: Computational Intelligence and Applications

Cognitive Computation Special Issue on Cognitive Behavioural Systems

Cognitive and Emotional Information Processing for Human–Machine Interaction

Special Issue on ICONIP 2012

Advances in Brain-Inspired Cognitive Systems

An Efficient Corpus-Based Stemmer

Similar Documents

Cognitive Computation: An Introduction

Computational Creativity, Intelligence and Autonomy

Sentic Computing

BLISS: an Artificial Language for Learnability Studies

Special Issue Editorial: Computational Intelligence and Applications

Cognitive Computation Special Issue on Cognitive Behavioural Systems

Cognitive and Emotional Information Processing for Human–Machine Interaction

Special Issue on ICONIP 2012

Advances in Brain-Inspired Cognitive Systems

An Efficient Corpus-Based Stemmer