vasp-5.4.4.pl2-vtst with nvc21.2-0OpenMPI3.1.5

[oschreiber]

Hi,
I have installed Nvidia's suite of PGI's compilers and libraries:
https://developer.nvidia.com/nvidia-hpc-sdk-downloads
$ wget https://developer.download.nvidia.com/h ... 1.2.tar.gz
It has BLAS, LAPACK, ScaLAPACK and OpenMPI3.1.5
It does not have FFT though except for cuda.
We want to rebuild:vasp-5.4.4.pl2-vtst with above.
I copied arch/makefile.include.linux_pgi
and edited it a little:

Code: Select all

oschreib@hpc2:/software/vasp-5.4.4.pl2-vtst/vasp-5.4.4.pl2-vtstnvc21.2-0OpenMPI3.1.5$ diff -wib arch/makefile.include.linux_pgi makefile.include
36,37c36,37
< LLIBS      += -L/opt/pgi/qd-2.3.17/install/lib -lqdmod -lqd
< INCS       += -I/opt/pgi/qd-2.3.17/install/include/qd
---
> LLIBS      += -L/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/lib -lqdmod -lqd
> INCS       += -I/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/include/qd
40,42c40,51
< FFTW       ?= /opt/pgi/fftw-3.3.4-PGI-16.10-OMPI-1.10.1
< LLIBS      += -L$(FFTW)/lib -lfftw3
< INCS       += -I$(FFTW)/include
---
> #FFTW       ?= /opt/pgi/fftw-3.3.4-PGI-16.10-OMPI-1.10.1
> #LLIBS      += -L$(FFTW)/lib -lfftw3
> #INCS       += -I$(FFTW)/include
> 
> #oschreib: using the gnu ones
> #FFTW       ?= /usr
> #LLIBS      += -L$(FFTW)/lib/x86_64-linux-gnu -lfftw3
> #INCS       += -I$(FFTW)/include
> 
> #these have undefined reference:
> #fftmpiw.o
> #fftw3d.o
80c89
< CUDA_ROOT  ?= /usr/local/cuda
---
> CUDA_ROOT  ?= /software/pgi//Linux_x86_64/21.2/cuda/11.2
88c97
< MPI_INC    = /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.1/include
---
> MPI_INC    = /software/pgi//Linux_x86_64/21.2/comm_libs/openmpi/openmpi-3.1.5/include
oschreib@hpc2:/software/vasp-5.4.4.pl2-vtst/vasp-5.4.4.pl2-vtstnvc21.2-0OpenMPI3.1.5$ 

But I get unresolved:

Code: Select all

/usr/bin/ld: fftmpiw.o: in function `fftbas_plan_mpi_':
/share/software/vasp-5.4.4.pl2-vtst/vasp-5.4.4.pl2-vtstnvc21.2-0OpenMPI3.1.5/build/std/fftmpiw.f90:158: undefined reference to `dfftw_plan_many_dft_

Thanks in advance for posting a better makefile.include

[merzuk.kaltak]

Hello,

The fftw library (not cufft from cuda) is required and must be linked to the correct path.
If you have edited the makefile.include after compilation to link to the proper libraries,
it is wise to perform a "make clean" and "make std gam ncl".
Just to make sure that all object files have been compiled with the correct libraries.

In case that doesn't work, it might be related to LD_LIBRARY_PATH, such that the linker doesn't find the corresponding files.
Assuming you install fftw to the path $FFTW and link with

Code: Select all

LLIBS      += -L$(FFTW)/lib -lfftw3
INCS       += -I$(FFTW)/include

then the path $(FFTW)/lib should contain the libfftw3.so library and $(FFTW)/include should have the following header files:

Code: Select all

-rw-r--r-- 1 kaltakm kaltakm 31394 Dec 10 21:38 fftw3.h
-rw-r--r-- 1 kaltakm kaltakm  2447 Dec 10 21:38 fftw3.f

If that doesn't work either, please submit a full error report as described in the forum guidelines.
Also, let us know with OS and which SDK version you are using.
FYI, currently only SDK-20.9 has been tested successfully.

[oschreiber]

Thank you very much Merrzuk.
I have done some of what you are advising. I am always careful about make clean clear when necessary.
I built 2 modules from fftw.org:
fftw/3.3.8nvc21.2OpenMPI3.1.5DP
fftw/3.3.8nvc21.2OpenMPI3.1.5SP
Here are the relevant portions of the makefile.inc: ( I am not sure of course)

Code: Select all

CPP_OPTIONS= -DHOST=\"LinuxPGI\" -DPGI16 \
             -DMPI -DMPI_BLOCK=8000 \
             -Duse_collective \
             -DscaLAPACK \
             -DCACHE_SIZE=4000 \
             -Davoidalloc \
             -Duse_bse_te \
             -Dtbdyn \
             -Duse_shmem \
             -DnoQuadPrecision
CPP        = pgfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)
FC         = mpif90
FCL        = mpif90 -lstdc++ -lgcc_s
FREE       = -Mfree
FFLAGS     = -Mnoupcase -Mbackslash
OFLAG      = -fast
OFLAG_IN   = $(OFLAG)
DEBUG      = -Mfree -O0 -traceback
# Use PGI provided BLAS and LAPACK libraries
BLAS       = -lblas
LAPACK     = -llapack
BLACS      =
SCALAPACK  = -lscalapack
LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)
# Software emulation of quadruple precision (COMMENTED OUT)
#LLIBS      += -L/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/lib -lqdmod -lqd
#INCS       += -I/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/include/qd
# Use the FFTs from fftw
FFTWS      ?= /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5SP
LLIBS      += -L$(FFTWS)/lib -lfftw3f_mpi -lfftw3f
INCS       += -I$(FFTWS)/include
FFTWD      ?= /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP
LLIBS      += -L$(FFTWD)/lib -lfftw3l_mpi -lfftw3l
INCS       += -I$(FFTWD)/include
OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o broyden.o
OBJECTS_O2 += fft3dlib.o pead.o
# For what used to be vasp.5.lib
CPP_LIB    = $(CPP)
FC_LIB     = pgfortran
CC_LIB     = pgcc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB   = -Mfree
OBJECTS_LIB= linpack_double.o getshmem.o
# For the parser library
CXX_PARS   = pgc++ --no_warnings
LIBS       += parser
LLIBS      += -Lparser -lparser -lstdc++
# Normally no need to change this
SRCDIR     = ../../src
BINDIR     = ../../bin
#================================================
# GPU Stuff
CPP_GPU    = -DCUDA_GPU -DRPROMU_CPROJ_OVERLAP -DCUFFT_MIN=28 -UscaLAPACK # -DUSE_PINNED_MEMORY 
OBJECTS_GPU= fftmpiw.o fftmpi_map.o fft3dlib.o fftw3d_gpu.o fftmpiw_gpu.o
CC         = pgcc
CXX        = pgc++
CFLAGS     = -fPIC -DADD_ -mp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY -DGPUSHMEM=300 -DHAVE_CUBLAS
CUDA_ROOT  ?= /software/pgi//Linux_x86_64/21.2/cuda/11.2
NVCC       := $(CUDA_ROOT)/bin/nvcc
CUDA_LIB   := -L$(CUDA_ROOT)/lib64 -lnvToolsExt -lcudart -lcuda -lcufft -lcublas
GENCODE_ARCH    := -gencode=arch=compute_30,code=\"sm_30,compute_30\" \
                   -gencode=arch=compute_35,code=\"sm_35,compute_35\" \
                   -gencode=arch=compute_60,code=\"sm_60,compute_60\"
MPI_INC    = /software/pgi/Linux_x86_64/21.2/comm_libs/openmpi/openmpi-3.1.5/include

ALSO!, I had to edit the .f90 to change the calls from dfftw_[...] to lfftw_[...]
VASP's source had dfftw whereas http://www.fftw.org's has lfftw_ names
Right now, I get a complete build but segmentation fault:
Total zeolite atoms = 6
[2. 2. 2. 2. 0. 0. 0.5 2.5 2.5 0.5 2. 2. 2. 2. 0. 0. 2. ]
Traceback (most recent call last):
File "/var/spool/slurmd/job13238/slurm_script", line 130, in <module>
energy = atoms.get_potential_energy() # Run vasp here
File "/home/gjw123/anaconda3/lib/python3.8/site-packages/ase/atoms.py", line 733, in get_potential_energy
energy = self._calc.get_potential_energy(self)
File "/home/gjw123/anaconda3/lib/python3.8/site-packages/ase/calculators/calculator.py", line 708, in get_potential_energy
energy = self.get_property('energy', atoms)
File "/home/gjw123/anaconda3/lib/python3.8/site-packages/ase/calculators/calculator.py", line 736, in get_property
self.calculate(atoms, [name], system_changes)
File "/home/gjw123/anaconda3/lib/python3.8/site-packages/ase/calculators/vasp/vasp.py", line 325, in calculate
raise calculator.CalculationFailed(
ase.calculators.calculator.CalculationFailed: vasp in . returned an error: 139

[merzuk.kaltak]

I advice to not change the pre-processed f90 files.

This has been tested for vasp-6.2.0 and not vasp-5.4.4, so I am not sure if the nvidia-sdk will work.

If you don't compile the old gpu-cuda port, then remove everything below "# GPU stuff" in your makefile.
Also, I suggest to use the nvfortran and nvc wrappers instead of the outdated pgfortran and pgcc commands from the SDK.
We compile fftw-3.3.8 with nvidia-hpc-sdk-20.9 with the following configure line after loading the modulefile "nvhpc-nompi" that is shipped with sdk-20.9 (no mpi support in fftw is compiled):

Code: Select all

./configure --enable-shared --enable-threads --enable-openmp --enable-avx2 --enable-avx --enable-sse2 --prefix=/path/to/install

Also, you should NOT link to the mpi-version of fftw but use only the serial libraries of fftw in your makefile.
The corresponding link line for including fftw is given in my previous post.

[oschreiber]

Wow, great actionable reply. Will apply carefully and report back!
Thanks!

[oschreiber]

Hi, I should have include OS, here it is:

Code: Select all

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"

Can you comment on single and double precision overall choice I have to make?
Also, what about these OBJECT lines? (should I comment them out?)

Code: Select all

OBJECTS    = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o broyden.o
OBJECTS_O2 += fft3dlib.o pead.o

Thanks!

[oschreiber]

Hi, I think one of the problems is with Single/Double precision:

Code: Select all

/share/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/build/std/fftmpiw.f90:158: undefined reference to `dfftw_plan_many_dft_'

The FFTW libraries have either l or f starting names:

Code: Select all

oschreib@hpc2:/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP$ nm /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/lib/libfftw3l.so|egrep _plan_many_dft_
000000000012cf10 T fftwl_plan_many_dft_c2r
000000000012d080 T fftwl_plan_many_dft_r2c
00000000001291f0 T lfftw_plan_many_dft_
000000000012a4e0 T lfftw_plan_many_dft__
0000000000129ab0 T lfftw_plan_many_dft_c2r_
000000000012ada0 T lfftw_plan_many_dft_c2r__
0000000000129660 T lfftw_plan_many_dft_r2c_
000000000012a950 T lfftw_plan_many_dft_r2c__
oschreib@hpc2:/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP$ nm /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5SP/lib/libfftw3f.so|egrep _plan_many_dft_
0000000000114810 T fftwf_plan_many_dft_c2r
0000000000114980 T fftwf_plan_many_dft_r2c
0000000000110af0 T sfftw_plan_many_dft_
0000000000111de0 T sfftw_plan_many_dft__
00000000001113b0 T sfftw_plan_many_dft_c2r_
00000000001126a0 T sfftw_plan_many_dft_c2r__
0000000000110f60 T sfftw_plan_many_dft_r2c_
0000000000112250 T sfftw_plan_many_dft_r2c__
oschreib@hpc2:/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP$ 

So that is why I have edited the .f90's to change the calls from dfftw_plan_many_dft to lfftw_plan_many_dft or sfftw_plan_many_dft
I get vasp_std, vasp_gam and vasp_ncl built but segmentation faults at execution which could be due to a mis-match or precision.
Thanks

[oschreiber]

It appears, I am not authorized to send attachments but the previous posts show the relevant areas of makefile.inc, displayed again here. Thanks!

Code: Select all

# Precompiler options
CPP_OPTIONS= -DHOST=\"LinuxPGI\" -DPGI16 \
             -Duse_collective \
             -DCACHE_SIZE=4000 \
             -Davoidalloc \
             -Duse_bse_te \
             -Dtbdyn \
             -Duse_shmem \
             -DnoQuadPrecision
CPP        = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)
FC         = nvfortran
FCL        = nvfortran -lstdc++ -lgcc_s
#FC         = mpif90
#FCL        = mpif90 -lstdc++ -lgcc_s
FREE       = -Mfree
FFLAGS     = -Mnoupcase -Mbackslash
OFLAG      = -fast
OFLAG_IN   = $(OFLAG)
DEBUG      = -Mfree -O0 -traceback
# Use PGI provided BLAS and LAPACK libraries
BLAS       = -lblas
LAPACK     = -llapack
BLACS      =
#SCALAPACK  = -lscalapack
LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)
# Software emulation of quadruple precision
#LLIBS      += -L/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/lib -lqdmod -lqd
#INCS       += -I/software/pgi//Linux_x86_64/21.2/compilers/extras/qd/include/qd
# Use the FFTs from fftw
#FFTWS      ?= /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5SP
#LLIBS      += -L$(FFTWS)/lib -lfftw3f
#INCS       += -I$(FFTWS)/include
FFTWD      ?= /software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP
LLIBS      += -L$(FFTWD)/lib -lfftw3l
INCS       += -I$(FFTWD)/include
OBJECTS    = fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o broyden.o
OBJECTS_O2 += fft3dlib.o pead.o
# For what used to be vasp.5.lib
CPP_LIB    = $(CPP)
FC_LIB     = nvfortran
CC_LIB     = nvc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB   = -Mfree
OBJECTS_LIB= linpack_double.o getshmem.o
# For the parser library
CXX_PARS   = nvc++ --no_warnings
LIBS       += parser
LLIBS      += -Lparser -lparser -lstdc++
# Normally no need to change this
SRCDIR     = ../../src
BINDIR     = ../../bin
#================================================

make[2]: Entering directory '/share/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/build/std'
nvfortran -lstdc++ -lgcc_s -o vasp c2f_interface.o base.o profiling.o openmp.o mpi.o mpi_shmem.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o xclib.o vdw_nl.o xclib_grad.o radial.o pseudo.o gridq.o ebs.o mkpoints.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symlib.o symmetry.o lattlib.o random.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o rhfatm.o hyperfine.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o pot.o dos.o elf.o tet.o tetweight.o hamil_rot.o bfgs.o dynmat.o  instanton.o  lbfgs.o sd.o   cg.o dimer.o bbm.o fire.o lanczos.o neb.o  qm.o opt.o chain.o dyna.o k-proj.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o tutor.o xml_writer.o brent.o stufak.o fileio.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o hamil_high.o nmr.o pead.o subrot.o subrot_scf.o paircorrection.o rpa_force.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o lcao_bare.o locproj.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o varpro.o minimax.o mlwf.o wnpr.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o acfdt.o chi.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o linear_response.o dmft.o auger.o dmatrix.o elphon.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -llapack -lblas -L/software/fftw/3.3.8/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/lib -lfftw3l -Lparser -lparser -lstdc++ 
/usr/bin/ld: fftw3d.o: in function `fftbas_':
/share/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/build/std/fftw3d.f90:77: undefined reference to `dfftw_plan_dft_3d_'
/usr/bin/ld: /share/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP/build/std/fftw3d.f90:81: undefined reference to `dfftw_execute_dft_'
[...]

[merzuk.kaltak]

Please do not use a "pgi" makefile for compilation with nvfortran.
The option "-DPGI16" shows me that you are doing this. This flag is used only for PGI-16.* compiler versions.
Please use "makefile.include.linux_nv" in combination with Nvidia-HPC-SDK.

Also, you can upload jpg or zip files only

[oschreiber]

Hi Merzuk,
Thanks for your reply.
>Please use "makefile.include.linux_nv" in combination with Nvidia-HPC-SDK.
Where can I find one?

Code: Select all

oschreib@hpc2:/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP$ ls -latr arch
-rw-r--r-- 1 oschreib rf 2470 Nov  4  2019 makefile.include.linux_pgi
-rw-r--r-- 1 oschreib rf 1091 Nov  4  2019 makefile.include.linux_intel_serial
-rw-r--r-- 1 oschreib rf 2099 Nov  4  2019 makefile.include.linux_intel
-rw-r--r-- 1 oschreib rf 2179 Nov  4  2019 makefile.include.linux_gnu
oschreib@hpc2:/software/vasp/5.4.4.pl2-vtst/ucdhpc-20.04/nvc21.2OpenMPI3.1.5DP$ 

>Also, you can upload jpg or zip files only
Where is the button to click on for uploading?
Thanks!