Error with mpirun VASP 6.3.2 with OpenACC+OpenMP
Posted: Fri Sep 30, 2022 4:10 am
Hi! I have compiled VASP 6.3.2 with OpenACC + OpenMP using the makefile.include.nvhpc_ompi_mkl_omp_acc shown below, and ran the 'make test' successfully. However, when I tried to run my job script shown below on a HPC allocation of 4 MPI ranks (1 rank per GPU) + 12 OpenMP threads per rank on a node with 48 core (24 cores per socket) AMD Milan 7413 + 4x Nvidia A100, it showed an error message indicated below. The mpirun command I used is as shown below. Hope to receive some advice on this issue. Thank you!
Job script
Output file
Makefile.include
Job script
Code: Select all
mpirun -np 4 --map-by ppr:2:socket:PE=12 --bind-to core \
-x OMP_NUM_THREADS=12 -x OMP_STACKSIZE=512m \
-x OMP_PLACES=cores -x OMP_PROC_BIND=close \
--report-bindings vasp_std
Code: Select all
----------------------------------------------------
OOO PPPP EEEEE N N M M PPPP
O O P P E NN N MM MM P P
O O PPPP EEEEE N N N M M M PPPP -- VERSION
O O P E N NN M M P
OOO P EEEEE N N M M P
----------------------------------------------------
running 4 mpi-ranks, with 12 threads/rank
distrk: each k-point on 1 cores, 4 groups
distr: one band on 1 cores, 1 groups
OpenACC runtime initialized ... 4 GPUs detected
vasp.6.3.2 27Jun22 (build Sep 28 2022 21:15:38) complex
POSCAR found type information on POSCAR LiMnNbO
POSCAR found : 4 types and 64 ions
Reading from existing POTCAR
scaLAPACK will be used selectively (only on CPU)
FATAL ERROR: data in update device clause was not found on device 4: name=lexch
file:/home/ljfong/VASP632/vasp.6.3.2/build/std/fock.f90 xc_fock_reader line:567
FATAL ERROR: data in update device clause was not found on device 3: name=lexch
file:/home/ljfong/VASP632/vasp.6.3.2/build/std/fock.f90 xc_fock_reader line:567
FATAL ERROR: data in update device clause was not found on device 1: name=lexch
file:/home/ljfong/VASP632/vasp.6.3.2/build/std/fock.f90 xc_fock_reader line:567
FATAL ERROR: data in update device clause was not found on device 2: name=lexch
file:/home/ljfong/VASP632/vasp.6.3.2/build/std/fock.f90 xc_fock_reader line:567
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[9289,1],3]
Exit code: 1
--------------------------------------------------------------------------
Code: Select all
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-D_OPENACC \
-DUSENCCL -DUSENCCLP2P
CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)
# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
FC = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.7 -mp
FCL = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.7 -mp -c++libs
FREE = -Mfree
FFLAGS = -Mbackslash -Mlarge_arrays
OFLAG = -fast
DEBUG = -Mfree -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda
# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = nvc++ --no_warnings
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself , change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -tp host
FFLAGS += $(VASP_TARGET_CPU)
# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')
# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /home/ljfong/VASP632/nvhpc
#NVVERSION = 22.7
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)
## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2
OFLAG_IN = -fast -Mwarperf
SOURCE_IN := nonlr.o
# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd
# Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
MKLROOT ?= /cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/imkl/2022.1.0
LLIBS_MKL = -Mmkl -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS += -I$(MKLROOT)/include/fftw
# Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
# Comment out the two lines below if you want to use scaLAPACK from MKL instead
#SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
#LLIBS_MKL = -L$(SCALAPACK_ROOT)/lib -lscalapack -Mmkl
LLIBS += $(LLIBS_MKL)
# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /home/ljfong/VASP632/hdf5
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /home/ljfong/VASP632/wannier/wannier90-3.1.0
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier
# For the fftlib library (hardly any benefit for the OpenACC GPU port, especially in combination with MKL's FFTs)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl