NDLI: Searching for Heavy Tails in Web Robot Traffic

Content Provider	IEEE Xplore Digital Library
Author	Doran, D. Gokhale, S.S.
Copyright Year	2010
Abstract	This paper presents a study on whether the heavy-tailed trends reported in Web traffic are present in the traffic generated by Web robots. The study is motivated by three factors: (i)~a significant volume of Web server traffic can now be attributed to Web robots, (ii)~the Web is continuing to evolve into a semantic and service-oriented environment where Web robots will play a central role, and (iii)~there are fundamental differences in the way robots and humans visit a site and search for information and these differences may lead to contrasts in the statistical patterns of the robots' requests compared to humans. We analyze Web robot traffic from a two-year access log from a Web server in the academic domain and study whether the response sizes, request inter-arrival times, and inter-session times exhibit heavy-tailed properties. In a multi-faceted analysis of the data we find that the response sizes and request inter-arrival times of robot requests do not exhibit heavy-tailed characteristics, contrasting the trends in these metrics in human traffic. However, we find that inter-session times of robots follow heavy-tailed characteristics similar to that of humans.
Starting Page	282
Ending Page	291
File Size	369421
Page Count	10
File Format	PDF
ISBN	9781424480821
DOI	10.1109/QEST.2010.42
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2010-09-15
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Measurement heavy-tailed distributions Humans Data models Weibull distribution Web server Web traffic Robots Web robot
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Property Analysis of Heavy-tailed Traffic by Simulator

Good news about heavy tails [network traffic]

Heavy-Tailed Probability Distributions in the World Wide Web (1998)

Fitting heavy-tailed HTTP traces with the new stratified EM-algorithm

Heavy-Tailed Probability Distributions in the World Wide Web (1998)

Characterizing Heavy-Tailed Distributions Induced by Retransmissions

Entropy-Based Heavy Tailed Distribution Transformation for Network Traffic Analysis

Using heavy-tailed distributions to stress-test kernel methods for segregating the firms that are likely to survive

Variable heavy tails in internet traffic (2004).

Searching for Heavy Tails in Web Robot Traffic

Similar Documents

Property Analysis of Heavy-tailed Traffic by Simulator

Good news about heavy tails [network traffic]

Heavy-Tailed Probability Distributions in the World Wide Web (1998)

Fitting heavy-tailed HTTP traces with the new stratified EM-algorithm

Heavy-Tailed Probability Distributions in the World Wide Web (1998)

Characterizing Heavy-Tailed Distributions Induced by Retransmissions

Entropy-Based Heavy Tailed Distribution Transformation for Network Traffic Analysis

Using heavy-tailed distributions to stress-test kernel methods for segregating the firms that are likely to survive

Variable heavy tails in internet traffic (2004).

Searching for Heavy Tails in Web Robot Traffic