- About EITEE
- Articles Online
- For Authors
- For Reviewers
- Features
- Subscription & Service
- Personal Center
- Logout
FOLLOWUS
- About EITEE
- Introduction
- Editorial Board
- Corresponding Experts
- Events
- Honors
- Contact Us
- Articles Online
- Current Issue
- Archive
- Special Issues
- Virtual Issues
- Highlights
- In Press
- For Authors
- Submit
- Call for papers
- Guide for Authors
- Ethics & Disclosures
- Fee & Funding
- Downloads
- For Reviewers
- Guide for Reviewers
- Ethics & Disclosures
- Thanks card
- Features
- FITEE Forum
- Engineering Front
- Paper Videos
- Editors’ Corner
- Other Resources
- Subscription & Service
- Subscription
- Services
Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal
- Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？
- Frontiers of Information Technology & Electronic Engineering Vol. 17, Issue 1, Pages: 15-31(2016)
- Affiliations：
Department of Computer Science and Engineering, NIT Silchar, India
- Author bio：
E-mail:dev.dipayan16@gmail.com
E-mail:ripon@cse.nits.ac.in
- Funds：
DOI：10.1631/FITEE.1500015
CLC： TP311Received：12 January 2015，
Accepted：11 June 2015，
Published：2016-01
- Accepted：
Scan QR Code
Dipayan DEV, Ripon PATGIRI. Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(1): 15-31.
Dipayan DEV, Ripon PATGIRI. Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(1): 15-31. DOI： 10.1631/FITEE.1500015.
摘要
目的
2
在这个“兆兆兆字节”（Exa byte）时代，数据量随时间指数率增长。剧增的数据在文件系统中制造了大量的元数据（metadata）。虽然Hadoop是处理大数据时最广泛采用的软件架构，其效率仍被研究者们广泛质疑。有必要为Hadoop创建一个有效且可扩展的元数据管理机制。
创新点
2
基于哈希的映射和子树分区适用于分布式元数据管理方案。基于哈希的映射在NameNode（Hadoop中存储元数据的服务器）间均衡地分配负载，但受到元数据空间局部性的限制；子树分区不需为保持负载均衡而迁移元数据，但也不能在服务器间均衡任务负载。本文提出一种称为DCMS（dynamic circular metadata splitting，动态环形元数据分割）的环形元数据管理机制（图3），并依此构建了Hadoop的改进框架—Dr. Hadoop（“Dr.”来自于本文作者名字首字母Dipayan DEV，Ripon PATGIRI）。NameNode是Hadoop的核心，其对所有文件路径树的保存失败将导致单点故障（single point of failure，SPoF）。DCMS能够移除Hadoop中的单点故障，从而提供一种有效且可扩展的元数据管理机制。
方法
2
通过使用局部保持哈希（locality-preserving hashing，LpH）保持元数据的空间局部性，通过使用一致性哈希（consistent hashing）保持服务器间的负载均衡，通过保留复制后的元数据实现高可靠性。
结论
2
理论分析表明，Dr. Hadoop架构在99.99%的时间能够可靠使用。通过衡量数据吞吐率、容错性和NameNode负载等性能，DCMS在大规模文件系统上较传统方法更具效力。
Abstract
In this Exa byte scale era
data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata
however
the efficiency of Hadoop is questioned numerous times by many researchers. Therefore
it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers
and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata
though it uniformly distributes the load among NameNodes
which are the metadata servers of Hadoop. In this paper
we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing
keeps replicated metadata for excellent reliability
and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files
failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.
关键词
Keywords
references
MK Aguilera , , , W Chen , , , S Toueg , , , 等 . . Heartbeat: atimeoutfree failure detector for quiescent reliable communication . . 1997 . . Proc. 11th Int. Workshop on Distributed Algorithms . . 126 - - 140 . . DOI: 10.1007/BFb0030680 http://doi.org/10.1007/BFb0030680 . .
Apache Software Foundation Hot Standby for NameNode , , 2012 . . Available from: http://issues.apache.org/jira/browse/HDFS-976 http://issues.apache.org/jira/browse/HDFS-976 . .
D Beaver , , , S Kumar , , , HC Li , , , 等 . . Finding aneedle in haystack: Facebookąŕs photo storage . . OSDI , , 2010 . . 47 - - 60 . . . .
D Biplob , , , S Sengupta , , , J Li , , , 等 . . FlashStore: highthroughput persistent key-value store . . 2010 . . Proc. VLDB Endowment . . 1414 - - 1425 . . . .
Bisciglia C Hadoop HA Configuration , , 2009 . . Available from: http://www.cloudera.com/blog/2009/07/22/hadoop-haconfiguration/ http://www.cloudera.com/blog/2009/07/22/hadoop-haconfiguration/ . .
RZPJ Braam . . Lustre: a Scalable, High PerformanceFile System , , 2007 . . : Inc : Cluster File Systems , , . .
SA Brandt , , , EL Miller , , , DDE Long , , , 等 . . Efficient metadata management in large distributed storage systems . . IEEE Symp. on Mass Storage Systems , , 2003 . . 290 - - 298 . . . .
Y Cao , , , C Chen , , , F Guo , , , 等 . . Es2: a cloud data storage system for supporting both OLTP and OLAP . . 2011 . . Proc. IEEE ICDE . . 291 - - 302 . . . .
PF Corbett , , , DG Feitelson . . The Vesta parallel file system . . ACM Trans. Comput. Syst , , 1996 . . 14 ( ( 3 ): ): 225 - - 264 . . DOI: 10.1145/233557.233558 http://doi.org/10.1145/233557.233558 . .
G DeCandia , , , D Hastorun , , , M Jampani , , , 等 . . Dynamo: Amazon’s highly available key-value store . . ACM SIGOPS Oper. Syst. Rev , , 2007 . . 41 ( ( 6 ): ): 205 - - 220 . . . .
D Dev , , , R Patgiri . . Performance evaluation of HDFS in big data management . . Int. Conf. on High Performance Computing and Applications , , 2014 . . 1 - - 7 . . . .
D Dev , , , R Patgiri . . HAR+: archive and metadatadistribution! Why not both? . . 2015 . . ICCCI . . in press . .
R Escriva , , , B Wong , , , EG Sirer , , , 等 . . HyperDex: a distributed, searchable key-value store . . ACM SIGCOMM Comput. Commun. Rev , , 2012 . . 42 ( ( 4 ): ): 25 - - 36 . . DOI: 10.1145/2377677.2377681 http://doi.org/10.1145/2377677.2377681 . .
H Fred , , , R McNab . . SimJava: a discrete event simulation library for Java . . Simul. Ser. , , 1998 . . 30 51 - - 56 . . . .
S Ghemawat , , , H Gobioff , , , ST Leung , , , 等 . . The Google file system . . 2003 . . Proc. 19th ACM Symp. on Operating Systems Principles . . 29 - - 43 . . . .
IF Haddad . . Pvfs: a parallel virtual file system for Linux clusters . . Linux J , , 2000 . . 5 . .
Wiki NameNode Failover, on Wiki ApacheHadoop , , 2012 . . Available from: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/NameNodeFailover . .
HDFS Hadoop AvatarNode High Availability , , 2010 . . Available from: http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html . .
D Karger , , , E Lehman , , , F Leighton , , , 等 . . Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web . . 1997 . . Proc.29th Annual ACM Symp. on Theory of Computing . . 654 - - 663 . . . .
S Kavalanekar , , , BL Worthington , , , Q Zhang , , , 等 . . Characterization of storage workload traces from production Windows Servers . . 2008 . . Proc. IEEE IISWC . . 119 - - 128 . . . .
D Lewin . . Consistent hashing and random trees: algorithms for caching in distributed networks . . Master Thesis, Department of EECS, MIT , , 1998 . . .
H Lim , , , B Fan , , , DG Andersen , , , 等 . . SILT:a memory-efficient, high-performance key-value store . . 2011 . . Proc. 23rd ACM Symp. on Operating SystemsPrinciples. . . . .
MK McKusick , , , S Quinlan . . GFS: evolution on fastforward . . ACM Queue , , 2009 . . 7 ( ( 7 ): ): 10 - - 20 . . DOI: 10.1145/1594204.1594206 http://doi.org/10.1145/1594204.1594206 . .
EL Miller , , , RH Katz . . Rama: an easy-to-use, highperformance parallel file system . . Parall. Comput , , 1997 . . 23 ( ( 4-5 ): ): 419 - - 446 . . DOI: 10.1016/S0167-8191(97)00008-2 http://doi.org/10.1016/S0167-8191(97)00008-2 . .
Miller, E.L., Greenan, K., Leung, A Reliable and efficient metadata storage and indexingusing nvram. , , 2008 . . Available from: dcslab.hanyang.ac.kr/nvramos08/EthanMiller.pdf. http://dcslab.hanyang.ac.kr/ nvramos08/EthanMiller.pdf. . .
D Nagle , , , D Serenyi , , , A Matthews . . The Panasas activescale storage cluster-delivering scalable high bandwidth storage. . . 2004 . . Proc. ACM/IEEE SC . . 1 - - 10 . . . .
E Okorafor , , , MK Patrick . . Availability of Jobtracker machine in hadoop/mapreduce zookeeper coordinated clusters . . Adv. Comput , , 2012 . . 3 ( ( 3 ): ): 19 - - 30 . . DOI: 10.5121/acij.2012.3302 http://doi.org/10.5121/acij.2012.3302 . .
JK Ousterhout , , , HD Costa , , , D Harrison , , , 等 . . Atrace-driven analysis of the Unix 4.2 BSD file system . . 1985 . . SOSP . . 15 - - 24 . . . .
I Raicu , , , IT Foste , , , P Beckman . . Making a casefor distributed file systems at exascale . . 2011 . . Proc. 3rdInt. Workshop on Large-Scale System and ApplicationPerformance . . 11 - - 18 . . DOI: 10.1145/1996029.1996034 http://doi.org/10.1145/1996029.1996034 . .
O Rodeh , , , A Teperman . . ZFS—a scalable distributedfile system using object disks . . 2003 . . IEEE Symp. on MassStorage Systems . . 207 - - 218 . . . .
M Satyanarayanan , , , JJ Kistler , , , P Kumar , , , 等 . . Coda: a highly available file system for a distributedworkstation environment . . IEEE Trans. Comput , , 1990 . . 39 ( ( 4 ): ): 447 - - 459 . . DOI: 10.1109/12.54838 http://doi.org/10.1109/12.54838 . .
K Shvachko , , , HR Kuang , , , S Radia , , , 等 . . TheHadoop Distributed File System . . 2010 . . IEEE 26th Symp. onMass Storage Systems and Technologies . . 1 - - 10 . . . .
Torodanhan Best Practice: DB2 High Availability Disaster Recovery , , 2009 . . Available from: http://www.ibm.com/developerworks/wikis/display/data/Best+Practice+-+DB2+High+Availability+Disaster+Recovery. http://www.ibm.com/developerworks/wikis/display/data/Best+Practice+-+DB2+High+Availability+Disaster+Recovery. . .
U.S . . Department of Commerce/NIST , , 1995 . . : VA : FIPS 180-1.Secure Hash Standard. National Technical InformationService, Springfield , , . .
F Wang , , , J Qiu , , , J Yang . . Hadoop highavailability through metadata replication . . 2009 . . Proc. 1stInt. Workshop on Cloud Data Management . . 37 - - 44 . . DOI: 10.1145/1651263.1651271 http://doi.org/10.1145/1651263.1651271 . .
SA Weil , , , KT Pollack , , , SA Brandt , , , 等 . . Dynamicmetadata management for petabyte-scale file systems . . 2004 . . SC . . 47 . .
SA Weil , , , SA Brandt , , , EL Miller , , , 等 . . CEPH:a scalable, high-performance distributed file system . . 2006 . . OSDI . . 307 - - 320 . . . .
T White . . Hadoop: the Definitive Guide , , 2009 . . O’Reilly Media, Inc. , , . .
BS White , , , M Walker , , , M Humphrey , , , 等 . . Legionfs: a secure and scalable file system supporting cross-domain highperformance applications . . 2001 . . Proc.ACM/IEEE Conf. on Supercomputing . . 59 . .
H Yadava . . The Berkeley DB Book , , 2007 . . : Apress : . .
Y Zhu , , , H Jiang , , , J Wang , , , 等 . . Hba: Distributed metadata management for large cluster-basedstorage systems . . IEEE Trans. Parall. Distrib. Syst , , 2008 . . 19 ( ( 6 ): ): 750 - - 763 . . DOI: 10.1109/TPDS.2007.70788 http://doi.org/10.1109/TPDS.2007.70788 . .
- L-PDFDownload
- Meta-XMLDownload
- BibTeXDownload
- EndnoteDownload
- RefMan Download
- NoteExpressDownload
- NoteFirstDownload
Publicity Resources
Related Articles
Related Author
Related Institution
Chat- Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
- Tel：+86-571-88273162 Email：fitee@zju.edu.cn
- Copyright © 2000 - 2025, Zhejiang University Press 京ICP备09064830号-19
京公网安备11010802024621 - It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
- Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
