NDLI: Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages

Please wait, while we are loading the content...

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages

Content Provider	ACM Digital Library
Author	Savoy, Jacques Dolamic, Ljiljana
Copyright Year	2010
Abstract	The main goal of this article is to describe and evaluate various indexing and search strategies for the Hindi, Bengali, and Marathi languages. These three languages are ranked among the world’s 20 most spoken languages and they share similar syntax, morphology, and writing systems. In this article we examine these languages from an Information Retrieval (IR) perspective through describing the key elements of their inflectional and derivational morphologies, and suggest a light and more aggressive stemming approach based on them. In our evaluation of these stemming strategies we make use of the FIRE 2008 test collections, and then to broaden our comparisons we implement and evaluate two language independent indexing methods: the $\textit{n}-gram$ and $trunc-\textit{n}$ (truncation of the first $\textit{n}$ letters). We evaluate these solutions by applying our various IR models, including the Okapi, Divergence from Randomness (DFR) and statistical language models (LM) together with two classical vector-space approaches: tf idf and $\textit{Lnu-ltc}.$ Experiments performed with all three languages demonstrate that the $I(n_{e})C2$ model derived from the Divergence from Randomness paradigm tends to provide the best mean average precision (MAP). Our own tests suggest that improved retrieval effectiveness would be obtained by applying more aggressive stemmers, especially those accounting for certain derivational suffixes, compared to those involving a light stemmer or ignoring this type of word normalization procedure. Comparisons between no stemming and stemming indexing schemes shows that performance differences are almost always statistically significant. When, for example, an aggressive stemmer is applied, the relative improvements obtained are ~28% for the Hindi language, ~42% for Marathi, and ~18% for Bengali, as compared to a no-stemming approach. Based on a comparison of word-based and language-independent approaches we find that the trunc-4 indexing scheme tends to result in performance levels statistically similar to those of an aggressive stemmer, yet better than the 4-gram indexing scheme. A query-by-query analysis reveals the reasons for this, and also demonstrates the advantage of applying a stemming or a trunc-4 indexing scheme.
Starting Page	1
Ending Page	24
Page Count	24
File Format	PDF
ISSN	15300226
e-ISSN	15583430
DOI	10.1145/1838745.1838748
Volume Number	9
Issue Number	3
Journal	ACM Transactions on Asian Language Information Processing (TALIP)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2010-09-01
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Bengali language Hindi language Indic languages Marathi language Natural language processing with Indo-European languages Search engines for Asian languages Stemmer
Content Type	Text
Resource Type	Article
Subject	Computer Science

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi and Bengali Languages (2010)

Comparative study of monolingual and multilingual search models for use with asian languages

A Fast Corpus-Based Stemmer

Discovering suffixes: A Case Study for Marathi Language

Discovering suffixes: a case study for marathi language.

Acquisition of Morphology of an Indic Language from Text Corpus

Achieving magnitude order improvement in Porter stemmer algorithm over multi-core architecture

Persian Language, Is Stemming Efficient?

A Rule-Based Subject-Correlated Arabic Stemmer

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages

Similar Documents

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi and Bengali Languages (2010)

Comparative study of monolingual and multilingual search models for use with asian languages

A Fast Corpus-Based Stemmer

Discovering suffixes: A Case Study for Marathi Language

Discovering suffixes: a case study for marathi language.

Acquisition of Morphology of an Indic Language from Text Corpus

Achieving magnitude order improvement in Porter stemmer algorithm over multi-core architecture

Persian Language, Is Stemming Efficient?

A Rule-Based Subject-Correlated Arabic Stemmer

Comparative Study of Indexing and Search Strategies for the Hindi, Marathi, and Bengali Languages