NDLI: The Ethicality of Web Crawlers

Please wait, while we are loading the content...

WI 2010 Cover Art

WI 2010 Title Page i

WI 2010 Title Page iii

WI 2010 Copyright Page

Welcome Message from the WI 2010 Conference Chair and Program Chair

WI 2010 Non-Program Committee Reviewers

Wikipedia and How to Use It for Semantic Document Representation

Game Theory for Security: Lessons Learned from Deployed Applications

The Sensor Web: Unpredictable, Noisy and Loaded with Errors

Issues on Preference-Modelling and Personalization in Information Retrieval

Why Robots Are More Than Just Agents

Bees Swarm Optimization Based Approach for Web Information Retrieval

Approximate Representations in the Medical Domain

Using Web-Intelligence for Excavating the Emerging Meaning of Target-Concepts

A Computational Cognitive Model for Simulating Heuristics Retrieval in Human Problem Solving

Stochastic Simulation of Web Users

Expertise Matching via Constraint-Based Optimization

Enabling Self-Organising Service Level Management with Automated Negotiation

Integrating Provenance Information in Reservoir Engineering

How to Improve Your Google Ranking: Myths and Reality

Image Set Classification Using Multi-layer Multiple Instance Learning with Application to Cannabis Website Classification

Towards Fully Distributed and Privacy-Preserving Recommendations via Expert Collaborative Filtering and RESTful Linked Data

Niche Product Retrieval in Top-N Recommendation

Language Models and Topic Models for Personalizing Tag Recommendation

Making the Most of a Web Search Session

Lightweight Clustering Methods for Webspam Demotion

MapMarker: Extraction of Postal Addresses and Associated Information for General Web Pages

CentralMatch: A Fast and Accurate Method to Identify Blog-Duplicates

Collaborative Web Search Utilizing Experts' Experiences

Multimodal Image Annotation Using Non-negative Matrix Factorization

Improving Collaborative Filtering in Social Tagging Systems for the Recommendation of Scientific Articles

Improving Privacy-Preserving NBC-Based Recommendations by Preprocessing

Towards a Self-Regulating Process of Pairwise Comparison in AHP

Ranking Approaches for Microblog Search

Group-Level Analysis by Extracting Semantic Relations from Query Graph

Predicting Web Search Hit Counts

On Using Query Logs for Static Index Pruning

Probabilistic Modeling of User-Generated Reviews

Let's Trust Users It is Their Search

Visual Content Structures for Wrapper Induction in Building Metasearch Systems

Relations Expansion: Extracting Relationship Instances from the Web

Using Web Technologies for Integrative Drug Discovery

An Efficient Method for Tagging a Query with Category Labels Using Wikipedia towards Enhancing Search Engine Results

Using Relational Similarity between Word Pairs for Latent Relational Search on the Web

Personalized Search Based on a User-Centered Recommender Engine

Reducing the Cold-Start Problem in Content Recommendation through Opinion Classification

Impact of One-Timer/N-Timer Object Classification on the Performance of Web Cache Replacement Algorithms

A Framework for Co-classification of Articles and Users in Wikipedia

WEST: Weighted-Edge Based Similarity Measurement Tools for Word Semantics

Linked Open Social Signals

Chem2Bio2RDF: A Linked Open Data Portal for Systems Chemical Biology

Query Answering in Fuzzy Description Logics with Data Type Support

F-Logic#: Loosely Coupling F-Logic Rules and Ontologies

Semi-automatic Acquisition of Semantic Descriptions of Processes in the Web

Ontology-Based Specific and Exhaustive User Profiles for Constraint Information Fusion for Multi-agents

Boosting Biomedical Entity Extraction by Using Syntactic Patterns for Semantic Relation Discovery

Learning a Large Scale of Ontology from Japanese Wikipedia

Semantic Graph Based Document-Indexing Strategy for MEDLINE

Learning in Presence of Ontology Mapping Errors

TOK: A Meta-model and Ontology for Heterogeneous Terminological, Linguistic and Ontological Knowledge Resources

SemPIF: A Semantic Meta-policy Interchange Format for Multiple Web Policies

A Hybrid Approach of Personalized Web Information Retrieval

User-Centered Evaluation Model for Ontology Selection

Domain-Specific Backlinking Services in the Web of Data

Towards Axiomatizing the Semantics of UML Activity Diagrams: A Situation-Calculus Perspective

A Scalable Indexing Mechanism for Ontology-Based Information Integration

Ranking Ontologies Using Verified Entities to Facilitate Federated Queries

The KGRAM Abstract Machine for Knowledge Graph Querying

Improving Diversity of Focused Summaries through the Negative Endorsements of Redundant Facts

Automatically Grouping Questions in Yahoo! Answers

Mining Fine Grained Opinions by Using Probabilistic Models and Domain Knowledge

Coauthor Network Topic Models with Application to Expert Finding

The Effects of Query Bursts on Web Search

Hierarchical Cost-Sensitive Web Resource Acquisition for Record Matching

A Hybrid Approach to Data Clustering Analysis with K-Means and Enhanced Ant-Based Template Mechanism

DSP: Robust Semi-supervised Dimensionality Reduction Using Dual Subspace Projections

Probabilistic Metrics for Soft-Clustering and Topic Model Validation

Tag Allocation Model: Model Noisy Social Annotations by Reason Finding

Text Clustering via Term Semantic Units

Organization of Information for the Web Using Hierarchical Fuzzy Clustering Algorithm Based on Co-occurrence Networks

Incorporating Seasonality into Search Suggestions Derived from Intranet Query Logs

Modeling Users' Information Goal Transitions and Satisfaction Judgment: Understanding the Full Search Process

Generating a Context-Aware Sentiment Lexicon for Aspect-Based Product Review Mining

Stakeholder Mining and Its Application to News Comparison

Parsing Publication Lists on the Web

ACO Based Approach and Integrating Information Retrieval Technologies in Selecting Bitmap Join Indexes

Exploring Special Items in Multimedia News Based on a Stakeholder Model

A Categorized Sentiment Analysis of Chinese Reviews by Mining Dependency in Product Features and Opinions from Blogs

Estimating News Coverage of Web Search Results

Product Feature Mining with Nominal Semantic Structure

Exploiting Puzzle Diversity in Puzzle Selection for ESP-Like GWAP Systems

Identifying Cohesive Subgroups and Their Correspondences in Multiple Related Networks

Characteristics Estimation of Information Sources by Information Diffusion Analysis

Predicting the Future with Social Media

Discovering Research Communities by Clustering Bibliographical Data

MQuery: Fast Graph Query via Semantic Indexing for Mobile Context

Modeling Ontology of Folksonomy with Latent Semantics of Tags

Communication Structure Discovery via Information Asymmetry in an Organizational Social Network

Asset-Mapping Approaches to Web-Based Collaborative Innovation

Investigating Homophily in Online Social Networks

Ant-Based Simulation of Opinion Spreading in Online Social Networks

Empirical Analysis and Multiple Level Views in Massive Social Networks

Topic Detection for Discussion Threads with Domain Knowledge

Effective Web Service Selection via Communities Formed by Super-Agents

Intelligent Agents System for Evacuation Processes Modeling

Feedback Dynamic Algorithms for Preemptable Job Scheduling in Cloud Systems

Web Service Selection for Multiple Agents with Incomplete Preferences

Business Process Decomposition Based on Service Relevance Mining

Learning Task Specific Web Services Compositions with Loops and Conditional Branches from Example Executions

Similarity-Based Bayesian Learning from Semi-structured Log Files for Fault Diagnosis of Web Services

Improving Web Service Survivability via Gracefully Degraded Substitution

Impacts of Analysts' Cognitive Styles on the Analytic Process

Mapping Features to Context Information: Supporting Context Variability for Context-Aware Pervasive Applications

Towards Understanding How Personality, Motivation, and Events Trigger Web User Activity

Web Intelligence in Tourism: User Modeling and Recommender System

An Approach to Model and Predict the Popularity of Online Contents with Explanatory Factors

Finding Resources and Collaborators within Digital Collections

Research Interests: Their Dynamics, Structures and Applications in Web Search Refinement

Visualizing Blog Archives to Explore Content- and Context-Related Interdependencies

Improving AbraQ: An Automatic Query Expansion Algorithm

Knowledgeable Explanations for Recommender Systems

An Adaptive Model for Probabilistic Sentiment Analysis

The Ethicality of Web Crawlers

Efficient and Correct Trust Propagation Using CloseLook

A Strong Authentication Protocol Based on Portable One-Time Dynamic URLs

Agent-Based Online Quality Measurement Approach in Cloud Computing Environment

Reachability Analysis in Privacy-Preserving Perturbed Graphs

SONAR: Towards User-centric Social Network Analysis and Visualization

Muzk Mesh: Interlinking Semantic Music Data

WI 2010 Author Index

WI 2010 Publisher's Information

The Ethicality of Web Crawlers

Content Provider	ACM Digital Library
Author	Sun, Yang Giles, C. Lee Councill, Isaac G.
Abstract	Search engines largely rely on web crawlers to collect information from the web. This has led to an enormous amount of web traffic generated by crawlers alone. To minimize negative aspects of this traffic on websites, the behaviors of crawlers may be regulated at an individual web server by implementing the Robots Exclusion Protocol in a file called “robots.txt”. Although not an official standard, the Robots Exclusion Protocol has been adopted to a greater or lesser extent by nearly all commercial search engines and popular crawlers. As many web site administrators and policy makers have come to rely on the informal contract set forth by the Robots Exclusion Protocol, the degree to which web crawlers respect robots.txt policies has become an important issue of computer ethics. In this research, we investigate and define rules to measure crawler ethics, referring to the extent to which web crawlers respect the regulations set forth in robots.txt configuration files. We test the behaviors of web crawlers in terms of ethics by deploying a crawler honeypot: a set of websites where each site is configured with a distinct regulation specification using the Robots Exclusion Protocol in order to capture specific behaviors of web crawlers.We propose a vector space model to represent crawler behavior and a set of models to measure the ethics of web crawlers based on their behaviors. The results show that ethicality scores vary significantly among crawlers. Most commercial web crawlers receive fairly low ethicality violation scores which means most of the crawlers’ behaviors are ethical; however, many commercial crawlers still consistently violate or misinterpret certain robots.txt rules.
Starting Page	668
Ending Page	675
Page Count	8
ISBN	9780769541914
DOI	10.1109/WI-IAT.2010.316
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2010-08-31
Access Restriction	Subscribed
Subject Keyword	Privacy Web crawler ethics Robots.txt Ethicality
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in