Scalability Challenges of an Industrial Implicit Finite Element Code

LS-DYNA is a well-known multiphysics code with both explicit and implicit time stepping capabilities. Implicit simulations rely heavily on sparse matrix computations, in particular direct solvers, and are notoriously much harder to scale than explicit simulations. In this paper, we investigate the scalability challenges of the implicit structural mode of LS-DYNA. In particular, we focus on linear constraint analysis, sparse matrix reordering, symbolic factorization, and numerical factorization. Our problem of choice for this study is a thermomechanical simulation of jet engine models built by Rolls-Royce with up to 200 million degrees of freedom, or equations. The models are used for engine performance analysis and design optimization, in particular optimization of tip clearances in the compressor and turbine sections of the engine. We present results using as many as 131,072 cores on the Blue Waters Cray XE6/XK7 supercomputer at NCSA and the Titan Cray XK7 supercomputer at OLCF. Since the main focus is on general linear algebra problems, this work is of interest for all linear algebra practitioners, not only developers of implicit finite element codes.