VASP6.3.2 forrtl: severe (174): SIGSEGV, segmentation fault occurred

[jeff_macdonald7]

I used instructions from here to build VASP https://www.intel.com/content/www/us/en ... ilers.html


lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

1 TB of RAM 10 GB of swap 128 cores.

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 4126170
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4126170
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

uname -a
Linux hostname 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux



head /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 1
model name : AMD EPYC 7773X 64-Core Processor
stepping : 2
microcode : 0xa001229
cpu MHz : 2195.894


SCRIPT_THAT_STARTS_MPIRUN & VASP6.3.2
#!/bin/bash
cores=126
executable='/var/tmp/intel_vasp_build/vasp.6.3.2/bin/vasp_std'
source /opt/intel/oneapi/setvars.sh
echo "Running VASP with executable at: "$executable
export OMP_NUM_THREADS=1
/usr/local/spack/opt/spack/linux-ubuntu20.04-zen3/aocc-3.2.0/openmpi-4.1.1-y2ogf3bkk3syj5oiz5ep554fs2kuvjsz/bin/mpirun -np $cores $executable 2>&1 | tee vasp_try${1}.out
wait

source /opt/intel/oneapi/setvars.sh

:: initializing oneAPI environment ...
bash: BASH_VERSION = 5.0.17(1)-release
args: Using "$@" for setvars.sh arguments:
:: advisor -- latest
:: ccl -- latest
:: clck -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: inspector -- latest
:: ipp -- latest
:: ippcp -- latest
:: ipp -- latest
:: itac -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vpl -- latest
:: vtune -- latest
:: oneAPI environment initialized ::

makefile.include started off as a copy of makefile.include.linux_intel

Makefile.include
CPP_OPTIONS = -DHOST=\"LinuxIFC\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dfock_dblbuf
CPP = fpp -f_com=no -free -w0 $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)
FC = mpiifort
FCL = mpiifort -mkl=sequential
FREE = -free -names lowercase
FFLAGS = -assume byterecl -w
OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/intel/oneapi/mkl/2023.0.0/interfaces/fftw3xf/libfftw3xf_intel.a
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = icc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
CXX_PARS = icpc
LLIBS = -lstdc++
VASP_TARGET_CPU ?= -march=core-avx2
FFLAGS += $(VASP_TARGET_CPU) -traceback -debug -g
FCL += -qmkl=sequential
MKLROOT ?= /opt/intel/oneapi/mkl/2023.0.0/lib/intel64

LLIBS += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
INCS =-I$(MKLROOT)/include/fftw


70% of the time MPIRUN & VASP crash with this error

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread-2.31.s 00007FADD5EA6420 Unknown Unknown Unknown
vasp_std 00000000008BF0D0 hamil_mp_hamiltmu 1626 hamil.F
vasp_std 00000000010BA5A9 david_mp_eddav_ 505 davidson.F
vasp_std 000000000115D063 elmin_ 499 electron.F
vasp_std 0000000001CB3E5F vamp_IP_electroni 5384 main.F
vasp_std 0000000001C8C1CD MAIN__ 3142 main.F
vasp_std 000000000040959D Unknown Unknown Unknown
libc-2.31.so 00007FADD5B75083 __libc_start_main Unknown Unknown
vasp_std 00000000004094BE Unknown Unknown Unknown
--------------------------------------------------------------------------


any help would be appreciated.

[martin.schlipf]

Does this happen for any system, even very small ones, or do you have one big system that you want to calculate?

[andreas.singraber]

Hello!

Please try setting

Code: Select all

ulimit -s unlimited

in the shell where your are trying to run VASP, maybe that solves the problem. Background: VASP allocates large amounts of memory on the stack. However, the default limit for stack allocations on Linux systems is often found to be only a few megabytes (8192 kB in your case, see output of ulimit -a). One can lift this restriction by using the command above. Please note that this change is not permanent and is lost as soon as you close the terminal. To make it permanent add the command to your .bashrc or comparable file.

Best,
Andreas Singraber

[jeff_macdonald7]

thanks andreas .. I updated /etc/security/limits. conf so stack size was unlimited.... will need to double / triple check my settings when I get back into work today.

[jeff_macdonald7]

stack size is unlimited, still SIGSEGV.

[jeff_macdonald7]

turns out I lied. PHD im working with had shell open for a long time, and updated /etc/security/limits.conf had no effect on his shell until
he logged out and logged in again. Testing again, this time with stacksize ulimited for sure.

[jeff_macdonald7]

setting stacksize to unlimited stopped vasp / mpirun from SIGSEGV'ing. Thanks so much for the help!!!

NOTE: its worth noting that VASP runs faster on AMD cpus when compiled with the intel's compiler than when compiled with AMDs compiler. Dont know why and totally unexpected result.