NDLI: A research on a defending policy against the Webcrawler's attack

Content Provider	IEEE Xplore Digital Library
Author	Wei Tong Xiaoyao Xie
Copyright Year	2009
Description	Author affiliation: Key Laboratory of Information and Computing, Science of Guizhou Province, Guizhou Normal University, Guiyang, China (Xiaoyao Xie) \|\| School of Computer Science and Technology, Guizhou University, Guiyang, China (Wei Tong)
Abstract	With the increasing of the amount of Internet information, there are different kinds of web crawlers fetching information from websites at anytime, anywhere, some of which are fetching information normally and some are attacking websites based upon application level and then causing the breakdown of servers, For a website, how to distinguish different kinds of crawlers effectively and accurately prevent bad crawlers' attack, or decrease crawlers' burden on the website during the rush hour becomes an important case, This paper based on a deep analysis of web crawler' s fetching process, summarized the web crawlers' threats on the security of websites and then Provide a trap policy and defend way based on robots.txt and the flux monitor, at last it is proved to be an effective strategy and method to protest the web site though experiments.
Starting Page	363
Ending Page	366
File Size	481982
Page Count	4
File Format	PDF
ISBN	9781424438839
DOI	10.1109/ICASID.2009.5276948
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2009-08-20
Publisher Place	China
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Delay effects Crawlers Access protocols Yarn Information analysis robots.txt Uniform resource locators security Robot kinematics webcrawler Internet Web server flux monitor Monitoring
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in