Storage wall for exascale supercomputing

Wei HU; Guang-ming LIU; Qiong LI; Yan-huang JIANG; Gui-lin CAI

doi:10.1631/FITEE.1601336

Your Location：

Home >

Browse articles >

Storage wall for exascale supercomputing

Regular Papers | Updated：2022-05-19

- Storage wall for exascale supercomputing
- 面向E级高性能计算存储墙问题研究
- Frontiers of Information Technology & Electronic Engineering Vol. 17, Issue 11, Pages: 1154-1175(2016)
- Affiliations：
  
  1College of Computer, National University of Defense Technology, Changsha 410073, China
  2National Supercomputer Center in Tianjin, Tianjin 300457, China
- Author bio：
  
  E-mail:huwei@nscc-tj.gov.cn
  E-mail:liugm@nscc-tj.gov.cn
  E-mail:qiong_joan_li@aliyun.com
  E-mail:yhjiang@nudt.edu.cn
  E-mail:cc_cai@163.com
- Funds：
- DOI：10.1631/FITEE.1601336
  CLC： TP338.6
- Received：15 June 2015，
  
  Accepted：05 October 2015，
  
  Published：2016-11
- Accepted：
Scan QR Code
Wei HU, Guang-ming LIU, Qiong LI, et al. Storage wall for exascale supercomputing[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(11): 1154-1175.
DOI：

Wei HU, Guang-ming LIU, Qiong LI, et al. Storage wall for exascale supercomputing[J]. Frontiers of Information Technology & Electronic Engineering, 2016, 17(11): 1154-1175. DOI： 10.1631/FITEE.1601336.

摘要

I/O性能与计算性能的不匹配是超级计算机从P级向E级发展的主要障碍之一。当前，I/O密集型应用迅速增加，其运行效率受到了I/O性能的制约。为进一步量化I/O性能瓶颈，分析其对大规模并行应用可扩展性的重要影响，本文在定义和分析存储受限加速比的基础上，第一次从并行应用可扩展性的角度提出"存储墙"的定义，量化和讨论了"存储墙"的存在性，并基于存储墙特性对现有高性能计算系统结构提出了新的分类方法和可扩展性分析方法。通过基于"天河1A"超级计算机的实验和基于"美洲虎"超级计算机的案例分析，本文验证了存储墙的存在性并分析了其变化特性。这些实验和分析结果对缓解和消除未来高性能计算机的存储墙瓶颈，实现平衡的软硬件设计具有重要指导意义。

Abstract

The mismatch between compute performance and I/O performance has long been a stumbling block as supercomputers evolve from petaflops to exaflops. Currently

many parallel applications are I/O intensive

and their overall running times are typically limited by I/O performance. To quantify the I/O performance bottleneck and highlight the significance of achieving scalable performance in peta/exascale supercomputing

in this paper

we introduce for the first time a formal definition of the 'storage wall' from the perspective of parallel application scalability. We quantify the effects of the storage bottleneck by providing a storage-bounded speedup

defining the storage wall quantitatively

presenting existence theorems for the storage wall

and classifying the system architectures depending on I/O performance variation. We analyze and extrapolate the existence of the storage wall by experiments on Tianhe-1A and case studies on Jaguar. These results provide insights on how to alleviate the storage wall bottleneck in system design and achieve hardware/software optimizations in peta/exascale supercomputing.

关键词

Keywords

references

S Agarwal , , , R Garg , , , MS Gupta , , , 等 . . Adaptive incremental checkpointing for massively parallel systems . . 2004 . . Proc. 18th Annual Int. Conf. on Supercomputing . . 277 - - 286 . . DOI: 10.1145/1006209.1006248 http://doi.org/10.1145/1006209.1006248 . .

T Agerwala . . Exascale computing: the challenges and opportunities in the next decade . . 2010 . . IEEE 16th Int. Symp. on High Performance Computer Architecture . . DOI: 10.1109/HPCA.2010.5416662 http://doi.org/10.1109/HPCA.2010.5416662 . .

SR Alam , , , JA Kuehn , , , RF Barrett , , , 等 . . Cray XT4: an early evaluation for petascale scientific simulation . . 2007 . . Proc. ACM/IEEE Conf. on Supercomputing . . 1 - - 12 . . DOI: 10.1145/1362622.1362675 http://doi.org/10.1145/1362622.1362675 . .

N Ali , , , PH Carns , , , K Iskra , , , 等 . . Scalable I/O forwarding framework for high-performance computing systems . . 2009 . . IEEE Int. Conf. on Cluster Computing and Workshops . . 1 - - 10 . . DOI: 10.1109/CLUSTR.2009.5289188 http://doi.org/10.1109/CLUSTR.2009.5289188 . .

GM Amdahl . . Validity of the single processor approach to achieving large scale computing capabilities . . 1967 . . Proc. Spring Joint Computer Conf. . . 483 - - 485 . . DOI: 10.1145/1465482.1465560 http://doi.org/10.1145/1465482.1465560 . .

J Bent , , , G Gibson , , , G Grider , , , 等 . . PLFS: a checkpoint file system for parallel applications . . 2009 . . Proc. Conf. on High Performance Computing Networking, Storage and Analysis . . 21 DOI: 10.1145/1654059.1654081 http://doi.org/10.1145/1654059.1654081 . .

F Cappello , , , A Geist , , , B Gropp , , , 等 . . Toward exascale resilience . . Int. J. High Perform. Comput. Appl. , , 2009 . . 23 ( ( 4 ): ): 374 - - 388 . . DOI: 10.1177/1094342009347767 http://doi.org/10.1177/1094342009347767 . .

P Carns , , , K Harms , , , W Allcock , , , 等 . . Understanding and improving computational science storage access through continuous characterization . . ACM Trans. Stor. , , 2011 . . 7 ( ( 3 ): ): 1 - - 26 . . DOI: 10.1145/2027066.2027068 http://doi.org/10.1145/2027066.2027068 . .

J Chen , , , YH Tang , , , Y Dong , , , 等 . . Reducing static energy in supercomputer interconnection networks using topology-aware partitioning . . IEEE Trans. Comput. , , 2016 . . 65 ( ( 8 ): ): 2588 - - 2602 . . DOI: 10.1109/TC.2015.2493523 http://doi.org/10.1109/TC.2015.2493523 . .

DE Culler , , , JP Singh , , , A Gupta . . Parallel Computer Architecture: a Hardware/Software Approach , , 1998 . . : San Francisco, USA : Morgan Kaufmann Publishers Inc. , , . .

IP Egwutuoha , , , D Levy , , , B Selic , , , 等 . . A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems . . J. Supercomput. , , 2013 . . 65 ( ( 3 ): ): 1302 - - 1326 . . DOI: 10.1007/s11227-013-0884-0 http://doi.org/10.1007/s11227-013-0884-0 . .

EN Elnozahy , , , JS Plank . . Checkpointing for peta-scale systems: a look into the future of practical rollback-recovery . . IEEE Trans. Depend. Secur. Comput. , , 2004 . . 1 ( ( 2 ): ): 97 - - 108 . . DOI: 10.1109/TDSC.2004.15 http://doi.org/10.1109/TDSC.2004.15 . .

EN Elnozahy , , , L Alvisi , , , YM Wang , , , 等 . . A survey of rollback-recovery protocols in message-passing systems . . ACM Comput. Surv. , , 2002 . . 34 ( ( 3 ): ): 375 - - 408 . . DOI: 10.1145/568522.568525 http://doi.org/10.1145/568522.568525 . .

M Fahey , , , J Larkin , , , J Adams . . I/O performance on a massively parallel cray XT3/XT4 . . 2008 . . IEEE Int. Symp. on Parallel and Distributed Processing . . 1 - - 12 . . DOI: 10.1109/IPDPS.2008.4536270 http://doi.org/10.1109/IPDPS.2008.4536270 . .

KB Ferreira , , , R Riesen , , , P Bridges , , , 等 . . Accelerating incremental checkpointing for extreme-scale computing . . Fut. Gener. Comput. Syst. , , 2014 . . 30 66 - - 77 . . DOI: 10.1016/j.future.2013.04.017 http://doi.org/10.1016/j.future.2013.04.017 . .

M Frasca , , , R Prabhakar , , , P Raghavan , , , 等 . . Virtual I/O caching: dynamic storage cache management for concurrent workloads . . 2011 . . Proc. Int. Conf. for High Performance Computing, Networking, Storage and Analysis . . 38 DOI: 10.1145/2063384.2063435 http://doi.org/10.1145/2063384.2063435 . .

T Gamblin , , , BR de Supinski , , , M Schulz , , , 等 . . Scalable load-balance measurement for SPMD codes . . 2008 . . Proc. ACM/IEEE Conf. on Supercomputing . . 1 - - 12 . . . .

JL Gustafson . . Reevaluating Amdahl's law . . Commun. ACM , , 1988 . . 31 ( ( 5 ): ): 532 - - 533 . . DOI: 10.1145/42411.42415 http://doi.org/10.1145/42411.42415 . .

PH Hargrove , , , JC Duell . . Berkeley lab checkpoint/restart (BLCR) for Linux clusters . . J. Phys. Conf. Ser. , , 2006 . . 46 ( ( 1 ): ): 494 - - 499 . . DOI: 10.1088/1742-6596/46/1/067 http://doi.org/10.1088/1742-6596/46/1/067 . .

JL Hennessy , , , DA Patterson . . Computer Architecture: a Quantitative Approach , , 2011 . . Elsevier , , . .

HPCwire DARPA Sets Ubiquitous HPC Program in Motion , , 2010 . . Available from: http://www.hpcwire.com/2010/08/10/darpa_sets_ubiquitous_hpc_program_in_motion/ http://www.hpcwire.com/2010/08/10/darpa_sets_ubiquitous_hpc_program_in_motion/ . .

W Hu , , , GM Liu , , , Q Li , , , 等 . . Storage speedup: an effective metric for I/O-intensive parallel application . . 2016 . . 18th Int. Conf. on Advanced Communication Technology . . 1 - - 12 . . DOI: 10.1109/ICACT.2016.7423395 http://doi.org/10.1109/ICACT.2016.7423395 . .

S Kalaiselvi , , , V Rajaraman . . A survey of checkpointing algorithms for parallel and distributed computers . . Sadhana , , 2000 . . 25 ( ( 5 ): ): 489 - - 510 . . DOI: 10.1007/BF02703630 http://doi.org/10.1007/BF02703630 . .

Y Kim , , , R Gunasekaran . . Understanding I/O workload characteristics of a peta-scale storage system . . J. Supercomput. , , 2015 . . 71 ( ( 3 ): ): 761 - - 780 . . DOI: 10.1007/s11227-014-1321-8 http://doi.org/10.1007/s11227-014-1321-8 . .

Y Kim , , , R Gunasekaran , , , GM Shipman , , , 等 . . Workload characterization of a leadership class storage cluster . . 2010 . . Petascale Data Storage Workshop . . 1 - - 5 . . DOI: 10.1109/PDSW.2010.5668066 http://doi.org/10.1109/PDSW.2010.5668066 . .

D Kotz , , , N Nieuwejaar . . Dynamic file-access characteristics of a production parallel scientific workload . . 1994 . . Proc. Supercomputing . . 640 - - 649 . . DOI: 10.1109/SUPERC.1994.344328 http://doi.org/10.1109/SUPERC.1994.344328 . .

WK Liao , , , A Ching , , , K Coloma , , , 等 . . Using MPI file caching to improve parallel write performance for large-scale scientific applications . . 2007 . . Proc. ACM/IEEE Conf. on Supercomputing . . 8 DOI: 10.1145/1362622.1362634 http://doi.org/10.1145/1362622.1362634 . .

N Liu , , , J Cope , , , P Carns , , , 等 . . On the role of burst buffers in leadership-class storage systems . . 2012 . . IEEE 28th Symp. on Mass Storage Systems and Technologies . . 1 - - 11 . . DOI: 10.1109/MSST.2012.6232369 http://doi.org/10.1109/MSST.2012.6232369 . .

Y Liu , , , R Gunasekaran , , , XS Ma , , , 等 . . Automatic identification of application I/O signatures from noisy server-side traces . . 2014 . . Proc. 12th USENIX Conf. on File and Storage Technologies . . 213 - - 228 . . . .

K Lu . . Research on Parallel File Systems Technology Toward Parallel Computing , , 1999 . . : Changsha, China : National University of Defense Technology , , PhD Thesis . .

R Lucas , , , J Ang , , , K Bergman , , , 等 . . DOE Advanced Scientific Computing Advisory Subcommittee (ASCAC) Report: Top Ten Exascale Research Challenges . . USDOE Office of Science , , 2014 . . DOI: 10.2172/1222713 http://doi.org/10.2172/1222713 . .

EL Miller , , , RH Katz . . Input/output behavior of supercomputing applications . . 1991 . . Proc. ACM/IEEE Conf. on Supercomputing . . 567 - - 576 . . DOI: 10.1145/125826.126133 http://doi.org/10.1145/125826.126133 . .

J Moreira , , , M Brutman , , , J Castano , , , 等 . . Designing a highly-scalable operating system: the blue Gene/L story . . 2006 . . Proc. ACM/IEEE Conf. on Supercomputing . . 53 - - 61 . . DOI: 10.1109/SC.2006.23 http://doi.org/10.1109/SC.2006.23 . .

RA Oldfield , , , S Arunagiri , , , PJ Teller , , , 等 . . Modeling the impact of checkpoints on next-generation systems . . 2007 . . 24th IEEE Conf. on Mass Storage Systems and Technologies . . 30 - - 46 . . DOI: 10.1109/MSST.2007.4367962 http://doi.org/10.1109/MSST.2007.4367962 . .

BK Pasquale , , , GC Polyzos . . A static analysis of I/O characteristics of scientific applications in a production workload . . 1993 . . Proc. ACM/IEEE Conf. on Supercomputing . . 388 - - 397 . . DOI: 10.1145/169627.169759 http://doi.org/10.1145/169627.169759 . .

JS Plank , , , M Beck , , , G Kingsley , , , 等 . . Libckpt: transparent checkpointing under Unix . . 1995 . . Proc. USENIX Technical Conf. . . 18 . .

A Purakayastha , , , C Ellis , , , D Kotz , , , 等 . . Characterizing parallel file-access patterns on a large-scale multiprocessor . . 1995 . . 9th Int. Parallel Processing Symp . . 165 - - 172 . . DOI: 10.1109/IPPS.1995.395928 http://doi.org/10.1109/IPPS.1995.395928 . .

J Sisilli . . Improved Solutions for I/O Provisioning and Application Acceleration . . 2015 . . Accessed on Nov. 18. 2015 Available from: http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/20150811_FD11_Sisilli.pdf http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/20150811_FD11_Sisilli.pdf . .

W Rudin . . Principles of Mathematical Analysis , , 1976 . . McGraw-Hill Publishing Co , , . .

J Shalf , , , S Dosanjh , , , J Morrison . . Exascale computing technology challenges . . 2011 . . 9th Int. Conf. on High Performance Computing for Computational Science . . 1 - - 25 . . DOI: 10.1007/978-3-642-19328-6_1 http://doi.org/10.1007/978-3-642-19328-6_1 . .

E Strohmaier , , , J Dongarra , , , H Simon , , , 等 . . TOP500 Supercomputer Sites . . 2015 . . Accessed on Dec. 30. 2015 Available from: http://www.top500.org/ http://www.top500.org/ . .

XH Sun , , , LM Ni . . Scalable problems and memorybounded speedup . . J. Parall. Distr. Comput. , , 1993 . . 19 ( ( 1 ): ): 27 - - 37 . . DOI: 10.1006/jpdc.1993.1087 http://doi.org/10.1006/jpdc.1993.1087 . .

University of California IOR HPC Benchmark . . 2007 . . Accessed on Sept. 1. 2014 Available from: http://sourceforge.net/projects/ior-sio/ http://sourceforge.net/projects/ior-sio/ . .

F Wang , , , Q Xin , , , B Hong , , , 等 . . File system workload analysis for large scale scientific computing applications . . 2004 . . Proc. 21st IEEE/12th NASA Goddard Conf. on Mass Storage Systems and Technologies . . 139 - - 152 . . . .

T Wang , , , S Oral , , , YD Wang , , , 等 . . Burstmem: a high-performance burst buffer system for scientific applications . . 2014 . . IEEE Int. Conf. on Big Data . . 71 - - 79 . . DOI: 10.1109/BigData.2014.7004215 http://doi.org/10.1109/BigData.2014.7004215 . .

T Wang , , , S Oral , , , M Pritchard , , , 等 . . Development of a burst buffer system for data-intensive applications.arXiv:1505.01765 , , 2015 . . Available from: http://arxiv.org/abs/1505.01765 http://arxiv.org/abs/1505.01765 . .

ZY Wang . . Reliability speedup: an effective metric for parallel application with checkpointing . . 2009 . . Int. Conf. on Parallel and Distributed Computing, Applications and Technologies . . 247 - - 254 . . DOI: 10.1109/PDCAT.2009.19 http://doi.org/10.1109/PDCAT.2009.19 . .

B Xie , , , J Chase , , , D Dillow , , , 等 . . Characterizing output bottlenecks in a supercomputer . . 2012 . . Int. Conf. for High Performance Computing, Networking, Storage and Analysis . . 1 - - 11 . . DOI: 10.1109/SC.2012.28 http://doi.org/10.1109/SC.2012.28 . .

XJ Yang , , , J Du , , , ZY Wang . . An effective speedup metric for measuring productivity in large-scale parallel computer systems . . J. Supercomput. , , 2011 . . 56 ( ( 2 ): ): 164 - - 181 . . DOI: 10.1007/s11227-009-0355-9 http://doi.org/10.1007/s11227-009-0355-9 . .

XJ Yang , , , ZY Wang , , , JL Xue , , , 等 . . The reliability wall for exascale supercomputing . . IEEE Trans. Comput. , , 2012 . . 61 ( ( 6 ): ): 767 - - 779 . . DOI: 10.1109/TC.2011.106 http://doi.org/10.1109/TC.2011.106 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Self-deployed execution environment for high performance computing

ONFS: a hierarchical hybrid file system based on memory, SSD, and HDD for high performance computers

Related Author

Mingtian SHAO

Kai LU

Wenzhe ZHANG

LIU Xin

LU Yu-tong

YU Jie

WANG Peng-fei

WU Jie-ting

Related Institution

College of Computer, National University of Defense Technology

School of Computer, National University of Defense Technology

Computer Science and Engineering, University of Nebraska-, Lincoln

National Supercomputing Center

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰