Successful installation of VASP 6.5.1 but failed test run

[karianaandrea_morenosader]

Hi,

I am running VASP on my login node in my lab's cloud and have successfully (I believe) installed all the numerical libraries and VASP 6.5.1. However, my test run failed and shows the following error:

BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 1356220 RUNNING AT login-2
= EXIT CODE: 11

I’m not sure if this happened because the allocated memory is insufficient, or if the numerical libraries or VASP were not properly installed. Please find attached the zip file with the makefile.include and the testsuite.log file containing the output.

Thank you very much
Best

[martin.schlipf]

I see two possible causes for the issue you observe:

1. Something went wrong with your MPI setup. If you look in the log of the testsuite, you'll see the following line

   Code: Select all

    running    1 mpi-ranks, with   64 threads/rank, on    1 nodes

   This should say that it is running on 4 mpi-ranks. This happens if the mpirun that you use is inconsistent with the mpif90 compiler. Please check

   Code: Select all

   which mpirun

   and make sure that it points to /usr/local/pkg/openmpi/4.1.3/bin/. You may also want to set

   Code: Select all

   export OMP_NUM_THREADS=1

   to eliminate another possible conflict. Generally, we recommend to default the threads to 1 (e.g. in the .bashrc file) and increase them afterwards and measure the performance as you change the value

2. Another common cause of segmentation faults is an insufficient stack size. Please type

   Code: Select all

   ulimit -s

   If that returns any other value than "unlimited", you will likely run into issues using VASP. Fortran program typically require much more space on the stack that C programs so the artificial separation into stack and heap limits the available memory. If you find that this is set to another value, please add

   Code: Select all

   ulimit -s unlimited

   to your .bashrc file.

[karianaandrea_morenosader]

Thank you very much, Martin!
I edited my .bashrc file accordingly, and the ulimit -s returns "unlimited". However, I still have a segmentation error.

Backtrace for this error:
[d-5-15-4:912544:0:912544] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x440000f8)
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Attached are the files with the errors. A few tests were successful, though.

**I should note that when I compiled VASP, I only compiled it for std, not for ncl or gam.

[martin.schlipf]

Thank you getting back to me. It seems that the obvious errors are fixed now, I will try to install VASP with your makefile.include and check whether I get the same error. One more thing you can try is

Code: Select all

ldd bin/vasp_std

and check whether the libraries all point to the ones you defined in the makefile.include. If that is not the case, you may need to adjust LD_LIBRARY_PATH so that these libraries are discovered during runtime.

[karianaandrea_morenosader]

Thank you very much! Here are all the libraries VASP_std is pointing

linux-vdso.so.1 (0x00007fff0fe77000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000152ef6ad2000)
libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x0000152ef4680000)
libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x0000152ef4636000)
libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000152ef435b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000152ef4274000)
libfftw3.so.3 => /lib/x86_64-linux-gnu/libfftw3.so.3 (0x0000152ef4059000)
libfftw3_threads.so.3 => /lib/x86_64-linux-gnu/libfftw3_threads.so.3 (0x0000152ef404f000)
libmpi_mpifh.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libmpi_mpifh.so.40 (0x0000152ef3feb000)
libmpi.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libmpi.so.40 (0x0000152ef3ecc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x0000152ef3eac000)
libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x0000152ef3e64000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000152ef3c3b000)
/lib64/ld-linux-x86-64.so.2 (0x0000152eff957000)
libopen-pal.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libopen-pal.so.40 (0x0000152ef3b80000)
libopen-rte.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libopen-rte.so.40 (0x0000152ef3ac1000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x0000152ef3a65000)
libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x0000152ef3a2e000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x0000152ef3a29000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x0000152ef3a0d000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x0000152ef39e3000)

Looks like the fftw3, openblas and gfortran is pointing to a different path. I have updated by .bashrc with the right paths, but when I do ldd vasp_std, I still see links to old paths for fftw3 and not for openblas. I believe I should rebuild vasp_std with the correct paths this time? what do you think?

[martin.schlipf]

Usually rebuilding should not be necessary. If you

Code: Select all

export LD_LIBRARY_PATH:/update/path:$LD_LIBRARY_PATH

you can change where the linker looks for libraries. It will look through those in order until it finds one.

[karianaandrea_morenosader]

Thanks for the reply! I just added this to my .bashrc and applied changes
export LD_LIBRARY_PATH=/update/path:$LD_LIBRARY_PATH
source ~/.bashrc

When I echo $LD_LIBRARY_PATH
I get the correct path, but vasp_std is still linked to the old one.

Not sure what else should I try

Thanks a lot

[martin.schlipf]

Sorry for the misunderstanding. You are supposed to update the path /update/path with the path where the libraries are.

[karianaandrea_morenosader]

Update: I figured out that I installed the static version of FFTW, which is why it could only find the .a files but not the .so files. Now, all the libraries point to the correct path...will give it a try to testsuite now

Thanks! Sorry, I actually did that, and this is why OpenBLAS is now linked to the correct path, which it wasn't before. However, I still have the problem with FFTW, which is linked to a different path

export LD_LIBRARY_PATH=/home/gridsan/kmoreno/scalapack/scalapack-2.2.2/lib:/home/gridsan/kmoreno/openblas/lib:/home/gridsan/kmoreno/fftw/lib:/usr/local/pkg/openmpi/4.1.3/lib:/home/gridsan/kmoreno/oscar/pkg/gaussian/g16/bsd:/home/gridsan/kmoreno/oscar/pkg/gaussian/g16:/home/gridsan/kmoreno/oscar/pkg/gaussian/gv/lib:$LD_LIBRARY_PATH

[karianaandrea_morenosader]

Hi Martin, I still have the same segmentation fault error. Here is a description of what I did:

1) I changed my barshrc to include this:
export PATH=/usr/local/pkg/openmpi/4.1.3/bin:$PATH
export LD_LIBRARY_PATH=/home/gridsan/kmoreno/scalapack/scalapack-2.2.2/lib:/home/gridsan/kmoreno/openblas/lib:/home/gridsan/kmoreno/fftw/lib:/usr/local/pkg/openmpi/4.1.3/lib:/home/gridsan/kmoreno/oscar/pkg/gaussian/g16/bsd:/home/gridsan/kmoreno/oscar/pkg/gaussian/g16:/home/gridsan/kmoreno/oscar/pkg/gaussian/gv/lib:$LD_LIBRARY_PATH
2) I did source ~/.bashrc
3) I verified that all libraries point out the right paths (now FFTW3 is fixed, along with OPENBLAS and openmpi) with ldd vasp_std
linux-vdso.so.1 (0x00007ffc44ab3000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000014e659b04000)
libopenblas.so.0 => /home/gridsan/kmoreno/openblas/lib/libopenblas.so.0 (0x000014e658b8b000)
libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x000014e658b41000)
libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x000014e658864000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014e65877d000)
libfftw3.so.3 => /home/gridsan/kmoreno/fftw/lib/libfftw3.so.3 (0x000014e65864b000)
libfftw3_threads.so.3 => /home/gridsan/kmoreno/fftw/lib/libfftw3_threads.so.3 (0x000014e658641000)
libmpi_mpifh.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libmpi_mpifh.so.40 (0x000014e6585dd000)
libmpi.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libmpi.so.40 (0x000014e6584be000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014e65849c000)
libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x000014e658454000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014e65822b000)
/lib64/ld-linux-x86-64.so.2 (0x000014e662989000)
libopen-pal.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libopen-pal.so.40 (0x000014e658170000)
libopen-rte.so.40 => /usr/local/pkg/openmpi/4.1.3/lib/libopen-rte.so.40 (0x000014e6580b1000)
libhwloc.so.15 => /lib/x86_64-linux-gnu/libhwloc.so.15 (0x000014e658053000)
libevent_core-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x000014e65801e000)
libevent_pthreads-2.1.so.7 => /lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x000014e658019000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x000014e657ffd000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x000014e657fd3000)

4. My slurm + testsuite.log file is attached. Note: My slurm has a line with export VASP_TESTSUITE_EXE_STD="mpirun -verbose -np 4 /home/gridsan/kmoreno/vasp/vasp.6.5.1/bin/vasp_std" but I also ran without it and still have the same error

Is there anything else I should try out?

[martin.schlipf]

I compared your makefile.include with our default setup and found a few differences.

Your setup (I split it over lines for consistency)

Code: Select all

LLIBS += -lstdc++
LLIBS += -L/usr/lib/x86_64-linux-gnu -lstdc++ 
LLIBS += -L$(OPENBLAS_ROOT)/lib -lopenblas -lgomp -lpthread -lm -ldl 
LLIBS += -L$(SCALAPACK_ROOT)/lib -lscalapack 
LLIBS += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_threads

Our setup

Code: Select all

LLIBS = -lstdc++
LLIBS += -L$(SCALAPACK_ROOT)/lib -lscalapack
LLIBS += -L$(OPENBLAS_ROOT)/lib -lopenblas
LLIBS += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp

[karianaandrea_morenosader]

Thanks, Martin. I will modify mine and check if the error is resolved. Just to confirm, if I make these changes, will they be reflected when I execute vasp_std, or should I rebuild it again?

[martin.schlipf]

For changes to the linking, you only need to redo the linking step. You do not need to recompile the whole code just running make should do the trick.

[karianaandrea_morenosader]

I made the changes to get my setup similar to yours, but I am still getting the same segmentation fault. I also realized that my library for scalapack is static and I wonder if this could cause the segmentation fault because it should be dynamic?
[d-17-10-2:909487:0:909487] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x440000f8)
[d-17-10-2:909488:0:909488] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x440000f8)
==== backtrace (tid: 909488) ====
0 /usr/local/pkg/ucx/1.14.0_cuda11.6/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x15461a513204]
1 /usr/local/pkg/ucx/1.14.0_cuda11.6/lib/libucs.so.0(+0x303ff) [0x15461a5133ff]
2 /usr/local/pkg/ucx/1.14.0_cuda11.6/lib/libucs.so.0(+0x306e6) [0x15461a5136e6]
3 /usr/local/pkg/openmpi/4.1.3/lib/libmpi.so.40(MPI_Comm_size+0x4) [0x154624d97144]
4 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xeccfed) [0x559cbc301fed]
5 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xecc5de) [0x559cbc3015de]
6 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xecc57d) [0x559cbc30157d]
7 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xa8b20) [0x559cbb4ddb20]
8 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xaa2ca) [0x559cbb4df2ca]
9 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xabadc) [0x559cbb4e0adc]
10 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0x76278b) [0x559cbbb9778b]
11 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0x7bee5f) [0x559cbbbf3e5f]
12 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xeb7d30) [0x559cbc2ecd30]
13 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xe9c580) [0x559cbc2d1580]
14 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0xeb8388) [0x559cbc2ed388]
15 /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x1546246b4d90]
16 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x1546246b4e40]
17 /home/gridsan/kmoreno/vasp/vasp.6.5.1/testsuite/../bin/vasp_std(+0x16dc5) [0x559cbb44bdc5]
=================================
==== backtrace (tid: 909487) ====

[martin.schlipf]

At this stage, I think it may be something with your library installation. Which OS are you using and do you have sudo rights? Please give me the version number as exact as possible so I can create a VM with the same setup.