NDLI: Minimizing the Network Overhead of Checkpointing in Cycle-harvesting Cluster Environments

Content Provider	IEEE Xplore Digital Library
Author	Nurmi, D. Brevik, J. Wolski, R.
Copyright Year	2005
Description	Author affiliation: Dept. of Comput. Sci., California Univ., Santa Barbara, CA (Nurmi, D.; Brevik, J.; Wolski, R.)
Abstract	Cycle-harvesting systems such as Condor have been developed to make desktop machines in a local area (which are often similar to clusters in hardware configuration) available as a compute platform. To provide a dual-use capability, opportunistic jobs harvesting cycles from the desktop must be checkpointed before the desktop resources are reclaimed by their owners and the job is evacuated. In this paper, we investigate a new system for computing efficient checkpoint schedules in cycle-harvesting environments. Our system records the historical availability from each resource and fits a statistical model to the observations. Because checkpointing must often traverse the network (i.e. the desktop hosts do not provide sufficient persistent storage for checkpoints), we combine this model with predictions of network performance to the storage site to compute a checkpoint schedule. When an application is initiated on a particular resource, the system uses the computed distribution to parameterize a Markov state-transition model for the application's execution, evaluates the expected time and network overhead as a function of the checkpoint interval, and numerically optimizes with respect to time. We report on the performance of and implementation of this system using the Condor cycle-harvesting environment at the University of Wisconsin. We also evaluate the efficiencies we achieve for a variety of network overheads using trace-based simulation. Finally, we validate our simulations against the observed performance with Condor. Our results indicate that while the choice of model distribution has a relatively small but positive effect on time efficiency, it has a substantial impact on network utilization
Starting Page	1
Ending Page	10
File Size	531571
Page Count	10
File Format	PDF
ISBN	0780394852
ISSN	15525244
DOI	10.1109/CLUSTR.2005.347074
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2005-09-27
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Checkpointing Computer networks Processor scheduling Workstations Hardware Availability Distributed computing Computational modeling Computer science Predictive models
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

The Effectiveness of Threshold-Based Scheduling Policies in BOINC Projects

Dynamic task scheduling in computing cluster environments

Models and modeling infrastructures for global computational platforms

Scheduling resources in multi-user, heterogeneous, computing environments with SmartNet

Resource policing to support fine-grain cycle stealing in networks of workstations

Performance modeling and prediction of nondedicated network computing

Agent-Based Autonomous Scheduling Mechanism Using Availability in Desktop Grid Systems

Checkpointing and Recovery Mechanism in Grid

Celebrating Diversity in Volunteer Computing

Minimizing the Network Overhead of Checkpointing in Cycle-harvesting Cluster Environments

Similar Documents

The Effectiveness of Threshold-Based Scheduling Policies in BOINC Projects

Dynamic task scheduling in computing cluster environments

Models and modeling infrastructures for global computational platforms

Scheduling resources in multi-user, heterogeneous, computing environments with SmartNet

Resource policing to support fine-grain cycle stealing in networks of workstations

Performance modeling and prediction of nondedicated network computing

Agent-Based Autonomous Scheduling Mechanism Using Availability in Desktop Grid Systems

Checkpointing and Recovery Mechanism in Grid

Celebrating Diversity in Volunteer Computing

Minimizing the Network Overhead of Checkpointing in Cycle-harvesting Cluster Environments