NDLI: Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

Content Provider	IEEE Xplore Digital Library
Author	Jin Xiong Yiming Hu Guojie Li Rongfeng Tang Zhihua Fan
Copyright Year	1990
Abstract	Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.
Sponsorship	IEEE Computer Society
Starting Page	803
Ending Page	816
Page Count	14
File Size	2517327
File Format	PDF
ISSN	10459219
Volume Number	22
Issue Number	5
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2011-05-01
Publisher Place	U.S.A.
Access Restriction	One Nation One Subscription (ONOS)
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Servers File systems Decision support systems Protocols Manuals Heuristic algorithms Reliability metadata management. Distributed file systems
Content Type	Text
Resource Type	Article
Subject	Signal Processing Computational Theory and Mathematics Hardware and Architecture

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Secure dynamic fragment and replica allocation in large-scale distributed file systems

Efficient and Scalable Metadata Management in EB-Scale File Systems

HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Supporting Scalable and Adaptive Metadata Management in Ultralarge-Scale File Systems

Implementing location independent invocation

A quorum-based protocol for searching objects in peer-to-peer networks

A virtual shared metadata storage for HDFS

QoS Driven Parallelization of Resources to Reduce File Download Delay

Semantic-Aware Metadata Organization Paradigm in Next-Generation File Systems

Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

Similar Documents

Secure dynamic fragment and replica allocation in large-scale distributed file systems

Efficient and Scalable Metadata Management in EB-Scale File Systems

HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems

Supporting Scalable and Adaptive Metadata Management in Ultralarge-Scale File Systems

Implementing location independent invocation

A quorum-based protocol for searching objects in peer-to-peer networks

A virtual shared metadata storage for HDFS

QoS Driven Parallelization of Resources to Reduce File Download Delay

Semantic-Aware Metadata Organization Paradigm in Next-Generation File Systems

Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems