Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？

Your Location：

Home >

Browse articles >

Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？

常规文章 | Updated：2022-05-19

- Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？
- Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal
- 信息与电子工程前沿（英文） 2016年17卷第1期页码：15-31
- Affiliations：
  
  Department of Computer Science and Engineering, NIT Silchar, India
- Author bio：
  
  E-mail:dev.dipayan16@gmail.com
  E-mail:ripon@cse.nits.ac.in
- Funds：
- DOI：10.1631/FITEE.1500015
  中图分类号： TP311
- 收稿：2015-01-12，
  
  录用：2015-6-11，
  
  纸质出版：2016-01
- Accepted：
Scan QR Code
Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？[J]. 信息与电子工程前沿（英文）, 2016,17(1):15-31.

Dipayan DEV, Ripon PATGIRI. Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(1): 15-31.
Dr. Hadoop: Hadoop的一种无限可扩展元数据管理机制—小象如何不老？[J]. 信息与电子工程前沿（英文）, 2016,17(1):15-31. DOI： 10.1631/FITEE.1500015.

Dipayan DEV, Ripon PATGIRI. Dr. Hadoop: an infinite scalable metadata management for Hadoop—How the baby elephant becomes immortal[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(1): 15-31. DOI： 10.1631/FITEE.1500015.

摘要

目的

2

在这个“兆兆兆字节”（Exa byte）时代，数据量随时间指数率增长。剧增的数据在文件系统中制造了大量的元数据（metadata）。虽然Hadoop是处理大数据时最广泛采用的软件架构，其效率仍被研究者们广泛质疑。有必要为Hadoop创建一个有效且可扩展的元数据管理机制。

创新点

2

基于哈希的映射和子树分区适用于分布式元数据管理方案。基于哈希的映射在NameNode（Hadoop中存储元数据的服务器）间均衡地分配负载，但受到元数据空间局部性的限制；子树分区不需为保持负载均衡而迁移元数据，但也不能在服务器间均衡任务负载。本文提出一种称为DCMS（dynamic circular metadata splitting，动态环形元数据分割）的环形元数据管理机制（图3），并依此构建了Hadoop的改进框架—Dr. Hadoop（“Dr.”来自于本文作者名字首字母Dipayan DEV，Ripon PATGIRI）。NameNode是Hadoop的核心，其对所有文件路径树的保存失败将导致单点故障（single point of failure，SPoF）。DCMS能够移除Hadoop中的单点故障，从而提供一种有效且可扩展的元数据管理机制。

方法

2

通过使用局部保持哈希（locality-preserving hashing，LpH）保持元数据的空间局部性，通过使用一致性哈希（consistent hashing）保持服务器间的负载均衡，通过保留复制后的元数据实现高可靠性。

结论

2

理论分析表明，Dr. Hadoop架构在99.99%的时间能够可靠使用。通过衡量数据吞吐率、容错性和NameNode负载等性能，DCMS在大规模文件系统上较传统方法更具效力。

Abstract

In this Exa byte scale era

data increases at an exponential rate. This is in turn generating a massive amount of metadata in the file system. Hadoop is the most widely used framework to deal with big data. Due to this growth of huge amount of metadata

however

the efficiency of Hadoop is questioned numerous times by many researchers. Therefore

it is essential to create an efficient and scalable metadata management for Hadoop. Hash-based mapping and subtree partitioning are suitable in distributed metadata management schemes. Subtree partitioning does not uniformly distribute workload among the metadata servers

and metadata needs to be migrated to keep the load roughly balanced. Hash-based mapping suffers from a constraint on the locality of metadata

though it uniformly distributes the load among NameNodes

which are the metadata servers of Hadoop. In this paper

we present a circular metadata management mechanism named dynamic circular metadata splitting (DCMS). DCMS preserves metadata locality using consistent hashing and locality-preserving hashing

keeps replicated metadata for excellent reliability

and dynamically distributes metadata among the NameNodes to keep load balancing. NameNode is a centralized heart of the Hadoop. Keeping the directory tree of all files

failure of which causes the single point of failure (SPOF). DCMS removes Hadoop’s SPOF and provides an efficient and scalable metadata management. The new framework is named ‘Dr. Hadoop’ after the name of the authors.

关键词

Keywords

references

MK Aguilera , , , W Chen , , , S Toueg , , , 等 . . Heartbeat: atimeoutfree failure detector for quiescent reliable communication . . 1997 . . Proc. 11th Int. Workshop on Distributed Algorithms . . 126 - - 140 . . DOI: 10.1007/BFb0030680 http://doi.org/10.1007/BFb0030680 . .

Apache Software Foundation Hot Standby for NameNode , , 2012 . . Available from: http://issues.apache.org/jira/browse/HDFS-976 http://issues.apache.org/jira/browse/HDFS-976 . .

D Beaver , , , S Kumar , , , HC Li , , , 等 . . Finding aneedle in haystack: Facebookąŕs photo storage . . OSDI , , 2010 . . 47 - - 60 . . . .

D Biplob , , , S Sengupta , , , J Li , , , 等 . . FlashStore: highthroughput persistent key-value store . . 2010 . . Proc. VLDB Endowment . . 1414 - - 1425 . . . .

Bisciglia C Hadoop HA Configuration , , 2009 . . Available from: http://www.cloudera.com/blog/2009/07/22/hadoop-haconfiguration/ http://www.cloudera.com/blog/2009/07/22/hadoop-haconfiguration/ . .

RZPJ Braam . . Lustre: a Scalable, High PerformanceFile System , , 2007 . . : Inc : Cluster File Systems , , . .

SA Brandt , , , EL Miller , , , DDE Long , , , 等 . . Efficient metadata management in large distributed storage systems . . IEEE Symp. on Mass Storage Systems , , 2003 . . 290 - - 298 . . . .

Y Cao , , , C Chen , , , F Guo , , , 等 . . Es2: a cloud data storage system for supporting both OLTP and OLAP . . 2011 . . Proc. IEEE ICDE . . 291 - - 302 . . . .

PF Corbett , , , DG Feitelson . . The Vesta parallel file system . . ACM Trans. Comput. Syst , , 1996 . . 14 ( ( 3 ): ): 225 - - 264 . . DOI: 10.1145/233557.233558 http://doi.org/10.1145/233557.233558 . .

G DeCandia , , , D Hastorun , , , M Jampani , , , 等 . . Dynamo: Amazon’s highly available key-value store . . ACM SIGOPS Oper. Syst. Rev , , 2007 . . 41 ( ( 6 ): ): 205 - - 220 . . . .

D Dev , , , R Patgiri . . Performance evaluation of HDFS in big data management . . Int. Conf. on High Performance Computing and Applications , , 2014 . . 1 - - 7 . . . .

D Dev , , , R Patgiri . . HAR+: archive and metadatadistribution! Why not both? . . 2015 . . ICCCI . . in press . .

R Escriva , , , B Wong , , , EG Sirer , , , 等 . . HyperDex: a distributed, searchable key-value store . . ACM SIGCOMM Comput. Commun. Rev , , 2012 . . 42 ( ( 4 ): ): 25 - - 36 . . DOI: 10.1145/2377677.2377681 http://doi.org/10.1145/2377677.2377681 . .

H Fred , , , R McNab . . SimJava: a discrete event simulation library for Java . . Simul. Ser. , , 1998 . . 30 51 - - 56 . . . .

S Ghemawat , , , H Gobioff , , , ST Leung , , , 等 . . The Google file system . . 2003 . . Proc. 19th ACM Symp. on Operating Systems Principles . . 29 - - 43 . . . .

IF Haddad . . Pvfs: a parallel virtual file system for Linux clusters . . Linux J , , 2000 . . 5 . .

Wiki NameNode Failover, on Wiki ApacheHadoop , , 2012 . . Available from: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/NameNodeFailover . .

HDFS Hadoop AvatarNode High Availability , , 2010 . . Available from: http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html . .

D Karger , , , E Lehman , , , F Leighton , , , 等 . . Consistent hashing and random trees: distributed caching protocols for relieving hot spots on the World Wide Web . . 1997 . . Proc.29th Annual ACM Symp. on Theory of Computing . . 654 - - 663 . . . .

S Kavalanekar , , , BL Worthington , , , Q Zhang , , , 等 . . Characterization of storage workload traces from production Windows Servers . . 2008 . . Proc. IEEE IISWC . . 119 - - 128 . . . .

D Lewin . . Consistent hashing and random trees: algorithms for caching in distributed networks . . Master Thesis, Department of EECS, MIT , , 1998 . . .

H Lim , , , B Fan , , , DG Andersen , , , 等 . . SILT:a memory-efficient, high-performance key-value store . . 2011 . . Proc. 23rd ACM Symp. on Operating SystemsPrinciples. . . . .

MK McKusick , , , S Quinlan . . GFS: evolution on fastforward . . ACM Queue , , 2009 . . 7 ( ( 7 ): ): 10 - - 20 . . DOI: 10.1145/1594204.1594206 http://doi.org/10.1145/1594204.1594206 . .

EL Miller , , , RH Katz . . Rama: an easy-to-use, highperformance parallel file system . . Parall. Comput , , 1997 . . 23 ( ( 4-5 ): ): 419 - - 446 . . DOI: 10.1016/S0167-8191(97)00008-2 http://doi.org/10.1016/S0167-8191(97)00008-2 . .

Miller, E.L., Greenan, K., Leung, A Reliable and efficient metadata storage and indexingusing nvram. , , 2008 . . Available from: dcslab.hanyang.ac.kr/nvramos08/EthanMiller.pdf. http://dcslab.hanyang.ac.kr/ nvramos08/EthanMiller.pdf. . .

D Nagle , , , D Serenyi , , , A Matthews . . The Panasas activescale storage cluster-delivering scalable high bandwidth storage. . . 2004 . . Proc. ACM/IEEE SC . . 1 - - 10 . . . .

E Okorafor , , , MK Patrick . . Availability of Jobtracker machine in hadoop/mapreduce zookeeper coordinated clusters . . Adv. Comput , , 2012 . . 3 ( ( 3 ): ): 19 - - 30 . . DOI: 10.5121/acij.2012.3302 http://doi.org/10.5121/acij.2012.3302 . .

JK Ousterhout , , , HD Costa , , , D Harrison , , , 等 . . Atrace-driven analysis of the Unix 4.2 BSD file system . . 1985 . . SOSP . . 15 - - 24 . . . .

I Raicu , , , IT Foste , , , P Beckman . . Making a casefor distributed file systems at exascale . . 2011 . . Proc. 3rdInt. Workshop on Large-Scale System and ApplicationPerformance . . 11 - - 18 . . DOI: 10.1145/1996029.1996034 http://doi.org/10.1145/1996029.1996034 . .

O Rodeh , , , A Teperman . . ZFS—a scalable distributedfile system using object disks . . 2003 . . IEEE Symp. on MassStorage Systems . . 207 - - 218 . . . .

M Satyanarayanan , , , JJ Kistler , , , P Kumar , , , 等 . . Coda: a highly available file system for a distributedworkstation environment . . IEEE Trans. Comput , , 1990 . . 39 ( ( 4 ): ): 447 - - 459 . . DOI: 10.1109/12.54838 http://doi.org/10.1109/12.54838 . .

K Shvachko , , , HR Kuang , , , S Radia , , , 等 . . TheHadoop Distributed File System . . 2010 . . IEEE 26th Symp. onMass Storage Systems and Technologies . . 1 - - 10 . . . .

Torodanhan Best Practice: DB2 High Availability Disaster Recovery , , 2009 . . Available from: http://www.ibm.com/developerworks/wikis/display/data/Best+Practice+-+DB2+High+Availability+Disaster+Recovery. http://www.ibm.com/developerworks/wikis/display/data/Best+Practice+-+DB2+High+Availability+Disaster+Recovery. . .

U.S . . Department of Commerce/NIST , , 1995 . . : VA : FIPS 180-1.Secure Hash Standard. National Technical InformationService, Springfield , , . .

F Wang , , , J Qiu , , , J Yang . . Hadoop highavailability through metadata replication . . 2009 . . Proc. 1stInt. Workshop on Cloud Data Management . . 37 - - 44 . . DOI: 10.1145/1651263.1651271 http://doi.org/10.1145/1651263.1651271 . .

SA Weil , , , KT Pollack , , , SA Brandt , , , 等 . . Dynamicmetadata management for petabyte-scale file systems . . 2004 . . SC . . 47 . .

SA Weil , , , SA Brandt , , , EL Miller , , , 等 . . CEPH:a scalable, high-performance distributed file system . . 2006 . . OSDI . . 307 - - 320 . . . .

T White . . Hadoop: the Definitive Guide , , 2009 . . O’Reilly Media, Inc. , , . .

BS White , , , M Walker , , , M Humphrey , , , 等 . . Legionfs: a secure and scalable file system supporting cross-domain highperformance applications . . 2001 . . Proc.ACM/IEEE Conf. on Supercomputing . . 59 . .

H Yadava . . The Berkeley DB Book , , 2007 . . : Apress : . .

Y Zhu , , , H Jiang , , , J Wang , , , 等 . . Hba: Distributed metadata management for large cluster-basedstorage systems . . IEEE Trans. Parall. Distrib. Syst , , 2008 . . 19 ( ( 6 ): ): 750 - - 763 . . DOI: 10.1109/TPDS.2007.70788 http://doi.org/10.1109/TPDS.2007.70788 . .

0

浏览量

35

Downloads

0

CSCD

>

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

相关文章

MDLB: a metadata dynamic load balancing mechanism based on reinforcement learning

Querying dynamic communities in online social networks

相关作者

武兆琪

卫今

张帆

郭威

谢光伟

Li WEIGANG

Edans F. O. SANDES

Jianya ZHENG

相关机构

国家数字交换系统工程技术研究中心

复旦大学计算机科学技术学院

复旦大学大数据研究院

Department of Computer Science, University of Brasilia

School of Computing, Staffordshire University, Stafford

Chat

地址：浙江杭州天目山路148号，浙江大学出版社邮编：310028
联系电话：+86-571-88273162 Email：fitee@zju.edu.cn
浙江大学出版社 © 2000—2025 京ICP备09064830号-19 京公网安备11010802024621
本系统建议在Chrome、 IE9+ 以上版本浏览器阅读本站内容，360浏览器请切换至极速模式
Cookies帮助我们提供服务并提供个性化体验。使用本网站，即表示您同意我们使用Cookies

⁰