NDLI: Automatic generation of question answer pairs from noisy case logs

Content Provider	IEEE Xplore Digital Library
Author	Ajmera, J. Joshi, S. Verma, A. Mittal, A.
Copyright Year	2014
Description	Author affiliation: Indian Inst. of Technol., New Delhi, New Delhi, India (Mittal, A.) \|\| IBM India Res. Lab., New Delhi, India (Ajmera, J.; Joshi, S.; Verma, A.)
Abstract	In a customer support scenario, a lot of valuable information is recorded in the form of `case logs'. Case logs are primarily written for future references or manual inspections and therefore are written in a hasty manner and are very noisy. In this paper, we propose techniques that exploit these case logs to mine real customer concerns or problems and then map them to well written knowledge articles for that enterprise. This mapping results into generation of question-answer (QA) pairs. These QA pairs can be used for a variety of applications such as dynamically updating the frequently-asked-questions (FAQs), updating the knowledge repository etc. In this paper we show the utility of these discovered QA pairs as training data for a question-answering system. Our approach for mining the case logs is based on a composite model consisting of two generative models, viz, hidden Markov model (HMM) and latent Dirichlet allocation (LDA) model. The LDA model explains the long-range dependencies across words due to their semantic similarity and HMM models the sequential patterns present in these case logs. Such processing results in crisp `problem statement' segments which are indicative of the real customer concerns. Our experiments show that this approach finds crisp problem-statements in 56% of the cases and outperforms other alternate methods for segmentation such as HMM, LDA and conditional random field (CRF). After finding these crisp problem-statements, appropriate answers are looked up from an existing knowledge repository index forming candidate QA pairs. We show that considering only the problemstatement segments for which the answers can be found further improves the segmentation performance to 82%. Finally, we show that when these QA pairs are used as training data, the performance of a question-answering system can be improved significantly.
Sponsorship	IEEE Comput. Soc.
Starting Page	436
Ending Page	447
File Size	594992
Page Count	12
File Format	PDF
ISBN	9781479925551
DOI	10.1109/ICDE.2014.6816671
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-03-31
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Hidden Markov models Semantics Noise measurement Syntactics Training Viterbi algorithm Context
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

A Hidden Markov Model for Automatic Generation of ER Diagrams from OWL Ontology

Hidden Markov Model training with side information

Noisy hidden Markov models for speech recognition

XML Structure Extraction from Plain Texts with Hidden Markov Model

Bearings-only tracking using hidden Markov models

Adjusted viterbi training for hidden markov models (2006).

Hidden Markov Models for automatic speech recognition

Improving denoising auto-encoder based speech enhancement with the speech parameter generation algorithm

A New Approach to Image Segmentation with Two-Dimensional Hidden Markov Models

Automatic generation of question answer pairs from noisy case logs

Similar Documents

A Hidden Markov Model for Automatic Generation of ER Diagrams from OWL Ontology

Hidden Markov Model training with side information

Noisy hidden Markov models for speech recognition

XML Structure Extraction from Plain Texts with Hidden Markov Model

Bearings-only tracking using hidden Markov models

Adjusted viterbi training for hidden markov models (2006).

Hidden Markov Models for automatic speech recognition

Improving denoising auto-encoder based speech enhancement with the speech parameter generation algorithm

A New Approach to Image Segmentation with Two-Dimensional Hidden Markov Models

Automatic generation of question answer pairs from noisy case logs