NDLI: Towards characterizing cloud backend workloads:insights from Google compute clusters

Please wait, while we are loading the content...

Towards characterizing cloud backend workloads:insights from Google compute clusters

Content Provider	ACM Digital Library
Author	Mishra, Asit K. Cirne, Walfredo Das, Chita R. Hellerstein, Joseph L.
Abstract	The advent of cloud computing promises highly available, efficient, and flexible computing services for applications such as web search, email, voice over IP, and web search alerts. Our experience at Google is that realizing the promises of cloud computing requires an extremely scalable backend consisting of many large compute clusters that are shared by application tasks with diverse service level requirements for throughput, latency, and jitter. These considerations impact (a) capacity planning to determine which machine resources must grow and by how much and (b) task scheduling to achieve high machine utilization and to meet service level objectives. Both capacity planning and task scheduling require a good understanding of task resource consumption (e.g., CPU and memory usage). This in turn demands simple and accurate approaches to workload classification-determining how to form groups of tasks (workloads) with similar resource demands. One approach to workload classification is to make each task its own workload. However, this approach scales poorly since tens of thousands of tasks execute daily on Google compute clusters. Another approach to workload classification is to view all tasks as belonging to a single workload. Unfortunately, applying such a coarse-grain workload classification to the diversity of tasks running on Google compute clusters results in large variances in predicted resource consumptions. This paper describes an approach to workload classification and its application to the Google Cloud Backend, arguably the largest cloud backend on the planet. Our methodology for workload classification consists of: (1) identifying the workload dimensions; (2) constructing task classes using an off-the-shelf algorithm such as k-means; (3) determining the break points for qualitative coordinates within the workload dimensions; and (4) merging adjacent task classes to reduce the number of workloads. We use the foregoing, especially the notion of qualitative coordinates, to glean several insights about the Google Cloud Backend: (a) the duration of task executions is bimodal in that tasks either have a short duration or a long duration; (b) most tasks have short durations; and (c) most resources are consumed by a few tasks with long duration that have large demands for CPU and memory.
Starting Page	34
Ending Page	41
Page Count	8
File Format	PDF
ISSN	01635999
DOI	10.1145/1773394.1773400
Journal	ACM SIGMETRICS Performance Evaluation Review (PERV)
Volume Number	37
Issue Number	4
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-01-10
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Content Type	Text
Resource Type	Article
Subject	Computer Networks and Communications Hardware and Architecture Software

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Towards characterizing cloud backend workloads: insights from google compute clusters (2010)

Optimized cloud placement of virtual clusters using biased importance sampling

Characterizing Web user sessions

Distributed Cloud Market:Who Benefits from Specification Flexibilities?

Towards Optimized Fine-Grained Pricing of IaaS Cloud Platform

Characterizing and interpreting periodic behavior in computer systems

On characterizing bandwidth requirements of parallel applications

Quantifying the Brown Side of Priority Schedulers:Lessons from Big Clusters

Measuring and characterizing home networks

Towards characterizing cloud backend workloads:insights from Google compute clusters

Similar Documents

Towards characterizing cloud backend workloads: insights from google compute clusters (2010)

Optimized cloud placement of virtual clusters using biased importance sampling

Characterizing Web user sessions

Distributed Cloud Market:Who Benefits from Specification Flexibilities?

Towards Optimized Fine-Grained Pricing of IaaS Cloud Platform

Characterizing and interpreting periodic behavior in computer systems

On characterizing bandwidth requirements of parallel applications

Quantifying the Brown Side of Priority Schedulers:Lessons from Big Clusters

Measuring and characterizing home networks

Towards characterizing cloud backend workloads:insights from Google compute clusters