NDLI: Unbinds data and tasks to improving the Hadoop performance

Content Provider	IEEE Xplore Digital Library
Author	Kun Lu Dong Dai Xuehai Zhou Mingming Sun Changlong Li Hang Zhuang
Copyright Year	2014
Description	Author affiliation: Comput. Sci., Texas Tech Univ., Lubbock, TX, USA (Dong Dai) \|\| Comput. Sci. & Technol, Univ. of Sci. & Technol. of China, Hefei, China (Kun Lu; Xuehai Zhou; Mingming Sun; Changlong Li; Hang Zhuang)
Abstract	Hadoop is a popular framework that provides easy programming interface of parallel programs to process large scale of data on clusters of commodity machines. Data intensive programs are the important part running on the cluster especially in large scale machine learning algorithm which executes of the same program iteratively. In-memory cache of input data is an efficient way to speed up these data intensive programs. However, we cannot be able to load all the data in memory because of the limitation of memory capacity. So, the key challenge is how we can accurately know when data should be cached in memory and when it ought to be released. The other problem is that memory capacity may even not enough to hold the input data of the running program. This leads to there is some data cannot be cached in memory. Prefetching is an effective method for such situation. We provide a unbinding technology which do not put the programs and data binded together before the real computation start. With unbinding technology, Hadoop can get a better performance when using caching and prefetching technology. We provide a Hadoop framework with unbinding technology named unbinding-Hadoop which decide the map tasks' input data in the map starting up phase, not at the job submission phase. Prefetching as well can be used in unbinding-Hadoop and can get better performance compared with the programs without unbinding. Evaluations on this system show that unbinding-Hadoop reduces the execution time of jobs by 40.2% and 29.2% with WordCount programs and K-means algorithm.
Starting Page	1
Ending Page	6
File Size	2994293
Page Count	6
File Format	PDF
ISBN	9781479956043
DOI	10.1109/SNPD.2014.6888710
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2014-06-30
Publisher Place	USA
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Prefetching cache system Software algorithms Clustering algorithms prefetch Programming unbinding Hard disks Electronic mail Acceleration
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers

Prefetching irregular references for software cache on cell

Data Cache Prefetching

The performance of runtime data cache prefetching in a dynamic optimization system

An Improved Adaptive Cache Prefetch Algorithm

When prefetching improves/degrades performance

Performance driven data cache prefetching in a dynamic software optimization system

Cache-miss-initiated prefetch in mobile environments.

Compiler support for software prefetching (1998).

Unbinds data and tasks to improving the Hadoop performance

Similar Documents

Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers

Prefetching irregular references for software cache on cell

Data Cache Prefetching

The performance of runtime data cache prefetching in a dynamic optimization system

An Improved Adaptive Cache Prefetch Algorithm

When prefetching improves/degrades performance

Performance driven data cache prefetching in a dynamic software optimization system

Cache-miss-initiated prefetch in mobile environments.

Compiler support for software prefetching (1998).

Unbinds data and tasks to improving the Hadoop performance