FTRP: a new fault tolerance framework using process replication and prefetching for high-performance computing

Wei HU; Guang-ming LIU; Yan-huang JIANG

doi:10.1631/FITEE.1601450

Your Location：

Home >

Browse articles >

FTRP: a new fault tolerance framework using process replication and prefetching for high-performance computing

Special Issue on Post-exascale Supercomputing | Updated：2022-05-19

- FTRP: a new fault tolerance framework using process replication and prefetching for high-performance computing
- 基于进程复制和预取的高性能计算容错框架
- Frontiers of Information Technology & Electronic Engineering Vol. 19, Issue 10, Pages: 1273-1290(2018)
- Affiliations：
  
  College of Computer, National University of Defense Technology, Changsha 410073, China
  National Supercomputer Center in Tianjin, Tianjin 300457, China
- Author bio：
  
  Wei HU, E-mail: huwei@nscc-tj.gov.cn
  [ "Guang-ming LIU, E-mail: liugm@nscc-tj.gov.cn" ]
  [ "Yan-huang JIANG, E-mail: yhjiang@nudt.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1601450
  CLC： TP338.6
- Received：03 August 2016，
  
  Revised：2018-10-;9，
  
  Published：2018-10
- Accepted：
Scan QR Code
Wei HU, Guang-ming LIU, Yan-huang JIANG. FTRP: a new fault tolerance framework using process replication and prefetching for high-performance computing[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(10): 1273-1290.
DOI：

Wei HU, Guang-ming LIU, Yan-huang JIANG. FTRP: a new fault tolerance framework using process replication and prefetching for high-performance computing[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(10): 1273-1290. DOI： 10.1631/FITEE.1601450.

摘要

随着超级计算机规模迅速增大，可靠性成为制约系统可用性的主要问题。现有容错机制，包括检查点技术和进程冗余等，不能有效解决该问题。为此，提出一种基于进程复制和预取的高性能计算容错框架-FTRP（fault tolerance frame work using process replication and prefetching），该框架兼具主动和被动容错机制的优点，引入创新的开销模型和主动容错机制，能够有效改善应用运行效率。提出"工作最多"（work-most，WM）的创新开销模型，基于故障预测结果和应用状态，从容错机制集中在线自适应给出运行容错决策。与程序运行过程中的局部性相似，我们第一次观察到超级计算机故障局部性现象。基于故障局部性，提出一种新的进程复制和进程预取相结合的容错机制，无论故障能否被预测到，都能够有效避免故障引起的损失。通过基于实际故障路径和普通故障预测准确率的模拟实验，并采用FTRP容错框架的应用，可以获得比现有容错机制10%的改进，且在P级甚至更大规模系统上有效。

Abstract

As the scale of supercomputers rapidly grows

the reliability problem dominates the system availability. Existing fault tolerance mechanisms

such as periodic checkpointing and process redundancy

cannot effectively fix this problem. To address this issue

we present a new fault tolerance framework using process replication and prefetching (FTRP)

combining the benefits of proactive and reactive mechanisms. FTRP incorporates a novel cost model and a new proactive fault tolerance mechanism to improve the application execution efficiency. The novel cost model

called the 'work-most' (WM) model

makes runtime decisions to adaptively choose an action from a set of fault tolerance mechanisms based on failure prediction results and application status. Similar to program locality

we observe the failure locality phenomenon in supercomputers for the first time. In the new proactive fault tolerance mechanism

process replication with process prefetching is proposed based on the failure locality

significantly avoiding losses caused by the failures regardless of whether they have been predicted. Simulations with real failure traces demonstrate that the FTRP framework outperforms existing fault tolerance mechanisms with up to 10% improvement in application efficiency for common failure prediction accuracy

and is effective for petascale systems and beyond.

关键词

Keywords

references

SR Alam , , , JA Kuehn , , , RF Barrett , , , 等 . . Cray XT4: an early evaluation for petascale scientific simulation . . Proc ACM/IEEE Conf on Supercomputing , , 2007 . . p.1 - - 12 . . DOI: 10.1145/1362622.1362675 http://doi.org/10.1145/1362622.1362675 . .

O Babaoglu , , , W Joy . . Converting a swap-based system to do paging in an architecture lacking page-referenced bits . . Proc 8 th ACM Symp on Operating Systems Principles , , 1981 . . p.78 - - 86 . . DOI: 10.1145/800216.806595 http://doi.org/10.1145/800216.806595 . .

A Bhatele , , , P Jetley , , , H Gahvari , , , 等 . . Architectural constraints to attain 1 exaflop/s for three scientific application classes . . Proc IEEE Int Parallel & Distributed Processing Symp , , 2011 . . p.80 - - 91 . . DOI: 10.1109/IPDPS.2011.18 http://doi.org/10.1109/IPDPS.2011.18 . .

MS Bouguerra , , , A Gainaru , , , LB Gomez , , , 等 . . Improving the computing efficiency of HPC systems using a combination of proactive and preventive checkpointing . . IEEE 27 th Int Symp on Parallel Distributed Processing , , 2013 . . p.501 - - 512 . . DOI: 10.1109/IPDPS.2013.74 http://doi.org/10.1109/IPDPS.2013.74 . .

D Brown , , , G Smith . . MPP2 Syslog Data (2006-2008) . . Technical Report, PNNL-SA-61371 , , 2008 . . .

F Cappello , , , H Casanova , , , Y Robert . . Checkpointing vs . . migration for post-petascale supercomputers. Proc 39 th Int Conf on Parallel Processing , , 2010 . . p.168 - - 177 . . DOI: 10.1109/ICPP.2010.26 http://doi.org/10.1109/ICPP.2010.26 . .

JT Daly . . A higher order estimate of the optimum checkpoint interval for restart dumps . . Fut Gener Comput Syst , , 2006 . . 22 ( ( 3 ): ): 303 - - 312 . . DOI: 10.1016/j.future.2004.11.016 http://doi.org/10.1016/j.future.2004.11.016 . .

PJ Denning . . The locality principle . . Commun ACM , , 2005 . . 48 ( ( 7 ): ): 19 - - 24 . . DOI: 10.1145/1070838.1070856 http://doi.org/10.1145/1070838.1070856 . .

C Dwork , , , N Lynch , , , L Stockmeyer . . Consensus in the presence of partial synchrony . . J ACM , , 1988 . . 35 ( ( 2 ): ): 288 - - 323 . . DOI: 10.1145/42282.42283 http://doi.org/10.1145/42282.42283 . .

IP Egwutuoha , , , D Levy , , , B Selic , , , 等 . . A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems . . J Supercomput , , 2013 . . 65 ( ( 3 ): ): 1302 - - 1326 . . DOI: 10.1007/s11227-013-0884-0 http://doi.org/10.1007/s11227-013-0884-0 . .

J Elliott , , , K Kharbas , , , D Fiala , , , 等 . . Combining partial redundancy and checkpointing for HPC . . IEEE 32 nd Int Conf on Distributed Computing Systems , , 2012 . . p.615 - - 626 . . DOI: 10.1109/ICDCS.2012.56 http://doi.org/10.1109/ICDCS.2012.56 . .

ENM Elnozahy , , , L Alvisi , , , YM Wang , , , 等 . . A survey of rollback-recovery protocols in message-passing systems . . ACM Comput Surv , , 2002 . . 34 ( ( 3 ): ): 375 - - 408 . . DOI: 10.1145/568522.568525 http://doi.org/10.1145/568522.568525 . .

M Fahey , , , J Larkin , , , J Adams . . I/O performance on a massively parallel Cray XT3/XT4 . . IEEE Int Symp on Parallel and Distributed Processing , , 2008 . . p.1 - - 12 . . DOI: 10.1109/IPDPS.2008.4536270 http://doi.org/10.1109/IPDPS.2008.4536270 . .

K Ferreira , , , J Stearley , , , JH Ⅲ Laros , , , 等 . . Evaluating the viability of process replication reliability for exascale systems . . Proc Int Conf for High Performance Computing, Networking, Storage and Analysis, Article 44 , , 2011 . . DOI: 10.1145/2063384.2063443 http://doi.org/10.1145/2063384.2063443 . .

A Gainaru , , , F Cappello , , , W Kramer . . Taming of the shrew: modeling the normal and faulty behaviour of large-scale HPC systems . . IEEE 26 th Int Symp on Parallel Distributed Processing , , 2012a . . p.1168 - - 1179 . . DOI: 10.1109/IPDPS.2012.107 http://doi.org/10.1109/IPDPS.2012.107 . .

A Gainaru , , , F Cappello , , , M Snir , , , 等 . . Fault prediction under the microscope: a closer look into HPC systems . . Proc Int Conf on High Performance Computing, Networking, Storage and Analysis, Article 77 , , 2012b . . DOI: 10.1109/SC.2012.57 http://doi.org/10.1109/SC.2012.57 . .

C George , , , S Vadhiyar . . ADFT: an adaptive framework for fault tolerance on large scale systems using application malleability . . Proc Comput Sci , , 2012 . . 9 166 - - 175 . . . .

C George , , , S Vadhiyar . . Fault tolerance on large scale systems using adaptive process replication . . IEEE Trans Comput , , 2015 . . 64 ( ( 8 ): ): 2213 - - 2225 . . DOI: 10.1109/TC.2014.2360536 http://doi.org/10.1109/TC.2014.2360536 . .

P Gujrati , , , Y Li , , , Z Lan , , , 等 . . A meta-learning failure predictor for blue gene/l systems . . Proc Int Conf on Parallel Processing , , 2007 . . p.1 - - 8 . . DOI: 10.1109/ICPP.2007.9 http://doi.org/10.1109/ICPP.2007.9 . .

S Gupta , , , P Xiang , , , Y Yang , , , 等 . . Locality principle revisited: a probability-based quantitative approach . . J Parall Distrib Comput , , 2013 . . 73 ( ( 7 ): ): 1011 - - 1027 . . DOI: 10.1016/j.jpdc.2013.01.010 http://doi.org/10.1016/j.jpdc.2013.01.010 . .

G Hamerly , , , C Elkan . . Bayesian approaches to failure prediction for disk drives . . Proc 18 th Int Conf on Machine Learning , , 2001 . . p.202 - - 209 . . . .

PH Hargrove , , , JC Duell . . Berkeley Lab Checkpoint/ Restart (BLCR) for Linux clusters . . J Phys Conf Ser , , 2006 . . 46 ( ( 1 ): ): 494 DOI: 10.1088/1742-6596/46/1/067 http://doi.org/10.1088/1742-6596/46/1/067 . .

JL Hellerstein , , , F Zhang , , , P Shahabuddin . . A statistical approach to predictive detection . . Comput Netw , , 2001 . . 35 ( ( 1 ): ): 77 - - 95 . . DOI: 10.1016/S1389-1286(00)00151-1 http://doi.org/10.1016/S1389-1286(00)00151-1 . .

W Hu , , , Y Jiang , , , G Liu , , , 等 . . DDC: Distributed Data Collection Framework for Failure Prediction in Tianhe Supercomputers . . Springer International Publishing , , 2015 . . p.18 - - 32 . . DOI: 10.1007/978-3-319-23216-4_2 http://doi.org/10.1007/978-3-319-23216-4_2 . .

S Kalaiselvi , , , V Rajaraman . . A survey of checkpointing algorithms for parallel and distributed computers . . Sadhana , , 2000 . . 25 ( ( 5 ): ): 489 - - 510 . . DOI: 10.1007/BF02703630 http://doi.org/10.1007/BF02703630 . .

Z Lan , , , Y Li . . Adaptive fault management of parallel applications for high-performance computing . . IEEE Trans Comput , , 2008 . . 57 ( ( 12 ): ): 1647 - - 1660 . . DOI: 10.1109/TC.2008.90 http://doi.org/10.1109/TC.2008.90 . .

Z Lan , , , J Gu , , , Z Zheng , , , 等 . . A study of dynamic metalearning for failure prediction in large-scale systems . . J Parall Distrib Comput , , 2010 . . 70 ( ( 6 ): ): 630 - - 643 . . DOI: 10.1016/j.jpdc.2010.03.003 http://doi.org/10.1016/j.jpdc.2010.03.003 . .

Y Liang , , , Y Zhang , , , M Jette , , , 等 . . Bluegene/l failure analysis and prediction models . . Int Conf on Dependable Systems and Networks , , 2006 . . p.425 - - 434 . . DOI: 10.1109/DSN.2006.18 http://doi.org/10.1109/DSN.2006.18 . .

CD Lu . . Scalable Diskless Checkpointing for Large Parallel Systems . . PhD Thesis, Champaign, IL, USA , , 2005 . . .

A Mohammed , , , R Kavuri , , , N Upadhyaya . . Fault tolerance: case study . . Proc 2 nd Int Conf on Computational Science, Engineering and Information Technology , , 2012 . . p.138 - - 144 . . DOI: 10.1145/2393216.2393240 http://doi.org/10.1145/2393216.2393240 . .

K Mohror , , , A Moody , , , BR de Supinski . . Asynchronous checkpoint migration with MRNet in the Scalable Checkpoint/Restart Library . . IEEE/IFIP Int Conf on Dependable Systems and Networks Workshops , , 2012 . . p.1 - - 6 . . DOI: 10.1109/DSNW.2012.6264668 http://doi.org/10.1109/DSNW.2012.6264668 . .

A Moody , , , G Bronevetsky , , , K Mohror , , , 等 . . Design, modeling, and evaluation of a scalable multi-level checkpointing system . . Proc ACM/IEEE Int Conf for High Performance Computing, Networking, Storage and Analysis , , 2010 . . p.1 - - 11 . . DOI: 10.1109/SC.2010.18 http://doi.org/10.1109/SC.2010.18 . .

E Pinheiro , , , WD Weber , , , LA Barroso . . Failure trends in a large disk drive population . . Proc 5 th USENIX Conf on File and Storage Technologies , , 2007 . . p.2 . .

JS Plank , , , M Beck , , , G Kingsley , , , 等 . . Libckpt: transparent checkpointing under Unix . . Proc USENIX Technical Conf Proc , , 1995 . . p.18 . .

JS Plank , , , K Li , , , MA Puening . . Diskless checkpointing . . IEEE Trans Parall Distrib Syst , , 1998 . . 9 ( ( 10 ): ): 972 - - 986 . . DOI: 10.1109/71.730527 http://doi.org/10.1109/71.730527 . .

E Roman . . A survey of checkpoint/restart implementations . . Technical Report LBNL-54942, Lawrence Berkeley National Laboratory , , 2002 . . .

RK Sahoo , , , AJ Oliner , , , I Rish , , , 等 . . Critical event prediction for proactive management in large-scale computer clusters . . Proc 9 th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining , , 2003 . . p.426 - - 435 . . DOI: 10.1145/956750.956799 http://doi.org/10.1145/956750.956799 . .

F Salfner , , , M Lenk , , , M Malek . . A survey of online failure prediction methods . . ACM Comput Surv , , 2010 . . 42 ( ( 3 ): ): 10.1 - - 10.42 . . DOI: 10.1145/1670679.1670680 http://doi.org/10.1145/1670679.1670680 . .

JC Sancho , , , F Petrini , , , G Johnson , , , 等 . . On the feasibility of incremental checkpointing for scientific computing . . Proc 18 th Int Symp on Parallel and Distributed Processing Symp , , 2004 . . p.58 DOI: 10.1109/IPDPS.2004.1302982 http://doi.org/10.1109/IPDPS.2004.1302982 . .

B Schroeder , , , E Pinheiro , , , WD Weber . . DRAM errors in the wild: a large-scale field study . . Proc 11 th Int Joint Conf on Measurement and Modeling of Computer Systems , , 2009 . . p.193 - - 204 . . DOI: 10.1145/2492101.1555372 http://doi.org/10.1145/2492101.1555372 . .

JS Vetter , , , F Mueller . . Communication characteristics of large-scale scientific applications for contemporary cluster architectures . . J Parall Distrib Comput , , 2003 . . 63 ( ( 9 ): ): 853 - - 865 . . DOI: 10.1016/S0743-7315(03)00104-7 http://doi.org/10.1016/S0743-7315(03)00104-7 . .

R Vilalta , , , S Ma . . Predicting rare events in temporal domains . . Proc IEEE Int Conf on Data Mining , , 2002 . . p.474 - - 481 . . DOI: 10.1109/ICDM.2002.1183991 http://doi.org/10.1109/ICDM.2002.1183991 . .

J Weinberg , , , MO McCracken , , , E Strohmaier , , , 等 . . Quantifying locality in the memory access patterns of HPC applications . . Proc ACM/IEEE Conf on Supercomputing , , 2005 . . p.50 DOI: 10.1109/SC.2005.59 http://doi.org/10.1109/SC.2005.59 . .

JW Young . . A first order approximation to the optimum checkpoint interval . . Commun ACM , , 1974 . . 17 ( ( 9 ): ): 530 - - 531 . . DOI: 10.1145/361147.361115 http://doi.org/10.1145/361147.361115 . .

Y Zhong , , , X Shen , , , C Ding . . Program locality analysis using reuse distance . . ACM Trans Program Lang Syst , , 2009 . . 31 ( ( 6 ): ): Article 20 DOI: 10.1145/1552309.1552310 http://doi.org/10.1145/1552309.1552310 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

OHTMA: an optimized heuristic topology-aware mapping algorithm on the Tianhe-3 exascale supercomputer prototype

Exploring high-performance processor architecture beyond the exascale

Moving from exascale to zettascale computing: challenges and techniques

Related Author

Xin-hai CHEN

Jie LIU

Bo YANG

Chun-ye GONG

Xin-biao GAN

Sheng-guo LI

Han XU

Yi-shui LI

Related Institution

Science and Technology on Parallel and Distributed Processing Laboratory, National University of Defense Technology

State Key Laboratory of Mathematical Engineering and Advanced Computing

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰