LS-DYNA: Installation and Usage on Caviness/DARWIN

LS-DYNA is an explicit dynamics solver that simulates the extreme materials deformation response of structures to periods of severe loading. It is a commercial product, so prior to using it on an IT RCI cluster a license must be purchased. A floating (network-based) license is required for use of LS-DYNA on clusters; the user must coordinate with his/her IT support staff to have a license server present on the UD campus network.

LS-DYNA is distributed for Windows and Linux operating systems. Many build variants exist for each release, varying by:

• Numerics: the entire program is built with either single- or double-precision floating point arithmetic
• Parallelism:
  • OpenMP: 1 process of 1 or more threads
  • MPP: 1 or more processes per node, 1 or more nodes
    • Intel MPI, Open MPI, or Platform MPI
  • Hybrid: 1 or more processes of 1 or more threads per node, 1 or more nodes
    • Intel MPI, Open MPI, or Platform MPI
• CPU optimizations: vector processing instructions
  • SSE2
  • AVX2
  • AVX512

Determining which version(s) and variant(s) are needed depends on the hardware present and the computational work being done. As of mid 2024, the current version is R15.0.2 and the best baseline variants to install on Caviness/DARWIN are the hybrid AVX2 variants in both single- and double-precision numerics.

For this example, LS-DYNA will be installed in a workgroup's storage. The base path $WORKDIR will be referenced in this recipe, which implies that prior to installation the workgroup command was used to spawn a workgroup shell. The recipe assumes version R15.0.2 and the variants mentioned above.

The LS-DYNA software packages must be downloaded to the cluster. To facilitate that, a directory must be created to hold those files:

$ mkdir -p "${WORKDIR}/sw/ls-dyna/attic/15.0.2"

(The -p ensures that any of the directories in that path that do not exist are created.) The files can be downloaded on a personal computer and uploaded to the directory just created, but they can also be downloaded directly on the cluster:

$ wget --directory-prefix="${WORKDIR}/sw/ls-dyna/attic/15.0.2" \
       --user=<USERNAME> --password=<PASSWORD> \
       '<DOWNLOAD-URL>'

The <USERNAME> and <PASSWORD> (as well as the download site URL) are provided when a license is purchased. The <DOWNLOAD-URL> is determined by browsing the download site and copying the link that would be clicked to download to a personal computer. For example, browsing and copying the link to download R15.0.2 for hybrid Open MPI with AVX2 instructions and single-precision numerics on Linux is

https://ftp.lstc.com/user/mpp-dyna/R15.0.2/linux/x86-64/ifort_190_avx2/HYB/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh

Downloading (or uploading) both packages, the attic directory now contains:

$ ls -l "${WORKDIR}/sw/ls-dyna/attic/15.0.2"
total 234420
-rw-r--r-- 1 user group 140454912 Apr  7 22:15 ls-dyna_hyb_d_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh
-rw-r--r-- 1 user group  99661023 Apr  7 22:15 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh

In preparation for installing these packages, a hierarchy of directories must be created to hold the variants of R15.0.2. Even though just two specific variants will be installed, the entire hierarchy is created:

• to promote a uniform organization between all versions of the software that get installed.
• to make it easy to add missing variants in the future.

The directory name matches the version, 15.0.2, with subdirectories for each option:

$ mkdir -p "${WORKDIR}/sw/ls-dyna/15.0.2/"{sse2,avx2,avx512}/{smp,hyb,mpp}/{double,single}

The single-precision variant is installed by changing to its subdirectory (created above) and running the appropriate installation extractor:

$ pushd "${WORKDIR}/sw/ls-dyna/15.0.2/avx2/hyb/single"
$ sh "${WORKDIR}/sw/ls-dyna/attic/15.0.2/ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.tgz_extractor.sh"

There will be several files in the directory, including the LS-DYNA executable:

$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a

Since the executable has a rather complicated name, a symlink is created so that the command ls-dyna is mapped to it:

$ ln -s ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405 ls-dyna
$ ls -l
total 154359
-rwxr-xr-x 1 user group  11402168 Mar 29 18:05 ansyscl
lrwxrwxrwx 1 user group        59 Jun  7 10:40 ls-dyna -> ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group 219543680 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405
-rwxr-xr-x 1 user group    437864 Mar 29 18:14 ls-dyna_hyb_s_R15_0_2_x64_centos79_ifort190_avx2_openmpi405.l2a
$ popd

This variant of LS-DYNA R15.0.2 is now installed, but there are underlying prerequisites to its usability: the Open MPI library on which it was built must also be made available. That step will be covered in a subsequent section.

For each additional variant of R15.0.2, the steps above are repeated with the appropriate modifications to the paths and file names. All hybrid and MPP variants will require the availability of their underlying MPI library.

As mentioned, the variants installed in this recipe are built atop Open MPI. As the file names indicate (and as documented in the R15.0.2 release notes), Open MPI 4.0.5 and the Intel 2019 compiler were used to build the two variants in question. Caviness/DARWIN have Intel 2019 compilers present, but the exact release of Open MPI built with that compiler may not be present. An instance of Open MPI 4.0.5 compiled with Intel 2019 should be produced for the workgroup's usage.

The procedure here will not include commentary, but is present as succinctly as possible. The process starts with setup of the directories and unpacking of the source code:

$ mkdir -p "${WORKDIR}/sw/openmpi/attic"
$ wget --directory-prefix="${WORKDIR}/sw/openmpi/attic" \
    https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.5.tar.bz2
$ mkdir -p "${WORKDIR}/sw/openmpi/src"
$ pushd "${WORKDIR}/sw/openmpi/src"
$ tar -xf "${WORKDIR}/sw/openmpi/attic/openmpi-4.0.5.tar.bz2"
$ cd openmpi-4.0.5
$ mkdir build-intel19-ls-dyna
$ cd build-intel19-ls-dyna
$ vpkg_rollback

For Caviness, the OFI libfabric communications library is used

$ vpkg_devrequire intel/2019 libfabric/1.17.1
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-lustre --with-io-romio-flags=--with-file-system=ufs+nfs+lustre \
        --with-libfabric="$LIBFABRIC_PREFIX" \
        CC=icc CXX=icpc FC=ifort

versus the OS-provided UCX library used on DARWIN

$ vpkg_devrequire intel/2019
$ ../configure --prefix="${WORKDIR}/sw/openmpi/4.0.5-intel19-ls-dyna" \
        --with-hwloc=/opt/shared/hwloc/2 \
        --with-pmix=/opt/shared/pmix/3 \
        --with-libevent=/usr \
        --with-lustre \
        --with-io-romio-flags="--with-file-system=ufs+nfs+lustre" \
        CC=icc CXX=icpc FC=ifort

If the configuration is successful, the software is built and installed (this will take some time):

$ make
$ make install
$ vpkg_rollback
$ popd