Linking error compiling GPU version of Vasp 6.3.0

[paulfons]

I just purchased a new Dell R750 server running centos 7.9.2009 with a Nvidia Ampere A30 card. I also have the Intel oneapi suite installed. I have compiled the cpu only version of Vasp without problem, however, I am having trouble compiling the graphics card version. I have moved makefile.include.nvhpc_ompi_mkl_omp_acc to makefile.include and modified the cuda version to 11.6 which is installed. When I try to run make, however, the following (several hundred lines worth) linking errors appear. Any suggestions as to what to try? Note that this is Vasp 6.3.0.


rsync -u ../../src/*.F ../../src/*.inc .
rm -f vasp ; make vasp ; cp vasp ../../bin/vasp_std
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/std'
mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -Mmkl -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -L/opt/wannier/lib -lwannier
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/lib/f90main.o: In function `main':
nvcYvyhbwf0ON1el.ll:(.text+0x2f): undefined reference to `MAIN_'
nccl2for.o: In function `nccl2for_mp_nccl2for_dummy_':
nccl2for.f90:(.text+0x4b): undefined reference to `for_write_seq_lis'
base.o: In function `base_mp_toreal_':
base.f90:(.text+0x51): undefined reference to `__qtod'
profiling.o: In function `profiling_mp_print_full_profile_':
profiling.f90:(.text+0xfb): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x13e): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x15e): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x1a0): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x1c0): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x1fd): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x235): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x278): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x2bb): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x3c9): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x3f9): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x41d): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x43f): undefined reference to `for_write_seq_fmt_xmit'

[marie-therese.huebsch]

Hi,

have you tried setting NVROOT manually? Perhaps the automatic search for NVROOT failed on your system. Please have a look at the following lines in your makefile.include:

Code: Select all

 58 # Specify your NV HPC-SDK installation (mandatory)
 59 #... first try to set it automatically
 60 NVROOT      =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')
 61 
 62 # If the above fails, then NVROOT needs to be set manually
 63 #NVHPC      ?= /opt/nvidia/hpc_sdk
 64 #NVVERSION   = 21.11
 65 #NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)

Comment out the automatic search and comment in the manual version. Then, please adjust the path information appropriate to your system and try compiling again. I hope this does the job.

Best regards,
Marie-Therese

[paulfons]

Thank you for the suggestion. I set the NVROOT manually and encountered the same linking errors. I have pasted the top of the output below. The path to the qdmod and qd libraries seems correct. I believe it was correct before as well. I installed the Nvidia library this afternoon using yum (centos 7.9) using the instructions on the Nvidia homepage. Does anyone have an idea as to how to fix the linking errors?

ls
libqd.a libqd_f_main.so libqd.la libqdmod.so libqd.so
libqd_f_main.a libqd_f_main.so.0 libqdmod.a libqdmod.so.0 libqd.so.0
libqd_f_main.la libqd_f_main.so.0.0.0 libqdmod.la libqdmod.so.0.0.0 libqd.so.0.0.0
(base) paulfons@kaon:/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib>ls -lt
total 1652
lrwxrwxrwx 1 root root 17 Mar 9 16:34 libqdmod.so.0 -> libqdmod.so.0.0.0
lrwxrwxrwx 1 root root 17 Mar 9 16:34 libqdmod.so -> libqdmod.so.0.0.0
lrwxrwxrwx 1 root root 21 Mar 9 16:34 libqd_f_main.so -> libqd_f_main.so.0.0.0
lrwxrwxrwx 1 root root 21 Mar 9 16:34 libqd_f_main.so.0 -> libqd_f_main.so.0.0.0
lrwxrwxrwx 1 root root 14 Mar 9 16:34 libqd.so -> libqd.so.0.0.0
lrwxrwxrwx 1 root root 14 Mar 9 16:34 libqd.so.0 -> libqd.so.0.0.0
-rwxr-xr-x 1 root root 1020 Feb 7 05:03 libqd_f_main.la
-rwxr-xr-x 1 root root 9312 Feb 7 05:03 libqd_f_main.so.0.0.0
-rw-r--r-- 1 root root 579118 Feb 7 05:03 libqdmod.a
-rwxr-xr-x 1 root root 992 Feb 7 05:03 libqdmod.la
-rwxr-xr-x 1 root root 198992 Feb 7 05:03 libqdmod.so.0.0.0
-rw-r--r-- 1 root root 565358 Feb 7 05:03 libqd.a
-rw-r--r-- 1 root root 3486 Feb 7 05:03 libqd_f_main.a
-rwxr-xr-x 1 root root 971 Feb 7 05:03 libqd.la
-rwxr-xr-x 1 root root 308136 Feb 7 05:03 libqd.so.0.0.0





mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -Mmkl -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -L/opt/wannier/lib -lwannier
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/lib/f90main.o: In function `main':
nvcYvyhbwf0ON1el.ll:(.text+0x2f): undefined reference to `MAIN_'
nccl2for.o: In function `nccl2for_mp_nccl2for_dummy_':
nccl2for.f90:(.text+0x4b): undefined reference to `for_write_seq_lis'
base.o: In function `base_mp_toreal_':
base.f90:(.text+0x51): undefined reference to `__qtod'
profiling.o: In function `profiling_mp_print_full_profile_':
profiling.f90:(.text+0xfb): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x13e): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x15e): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x1a0): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x1c0): undefined reference to `for_write_seq_fmt_xmit'
profiling.f90:(.text+0x1fd): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x235): undefined reference to `for_write_seq_lis'
profiling.f90:(.text+0x278): undefined reference to `for_write_seq_fmt'
profiling.f90:(.text+0x2bb): undefined reference to `for_write_seq_fmt'
much more ...

[marie-therese.huebsch]

Based on the missing routines, maybe this discussion from another forum helps: http://icl.cs.utk.edu/lapack-forum/view ... ?f=2&t=747
Could you try including libifcore?

[paulfons]

I manually ran the linker again right after the above failure adding "-lifcore" at the end of the linking sequence only to find a new error was encountered when trying to link "error adding symbols: DSO missing from command line" Is there a specific place I should add libifcore to the makefile?


(base) paulfons@kaon:/data/Software/Vasp/vasp.6.3.0/build/std>mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -Mmkl -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -L/opt/wannier/lib -lwannier -lifcore
/usr/bin/ld: minimax_functions2D.o: undefined reference to symbol '__fminq'
/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin/libimf.so: error adding symbols: DSO missing from command line
pgacclnk: child process exit status 1: /usr/bin/ld

[marie-therese.huebsch]

Did you locate libifcore.a on your system? Then, you can add it to the makefile.include in the following way:

Code: Select all

LIBIFCORE_PATH  ?= /path/to/your/libifcore/installation
LLIBS       += -L$(LIBIFCORE_PATH) -lifcore

Please let me know if this works.

[paulfons]

Thank you for your suggestion. I added the -lifcore to the LLIBS line and found the same results as in the last entry I made, namely a complaint about an undefined symbol __fminq and a statement that error adding symbols: DSO missing from command line
pgacclnk: child process exit status 1: /usr/bin/ld

Any further ideas? I am really curious to see the throughput with the Ampere card. A few years ago, I compiled the Cuda version of Vasp 5.4 with a toy GPU card and had no problems. In any case, let me thank you again for your suggestions.

make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/std'
mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -mp -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -lifcore -Mmkl -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -L/opt/wannier/lib -lwannier
/usr/bin/ld: minimax_functions2D.o: undefined reference to symbol '__fminq'
/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin/libimf.so: error adding symbols: DSO missing from command line
pgacclnk: child process exit status 1: /usr/bin/ld
make[2]: *** [vasp] Error 2
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
cp: cannot stat ‘vasp’: No such file or directory
make[1]: *** [all] Error 1
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
make: *** [std] Error 2

[marie-therese.huebsch]

Unfortunately, we can not reproduce this linking issue. The same toolchain/makefile.include runs smoothly for us on centos 8.3. So let us keep trying on your system. Here are a couple of suggestions:

Perhaps the order of linking causes an issue. Try to switch the ordering to the following:

Code: Select all

# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
MKLROOT    ?= /path/to/your/mkl/installation
LLIBS      += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS       += -I$(MKLROOT)/include/fftw

# Software emulation of quadruple precsion (mandatory)
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

Next, you may try to use this order for the mkl part:

Code: Select all

LLIBS      += -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -Mmkl

Could you maybe share what toolchain you used for the CPU version?
And could you please try to compile without mkl makefile.include.nvhpc_acc?

Best regards,
Marie-Therese

[paulfons]

Thank you for your comments. I have tried to use the non MKL version of the makefile.include file (makefile.include.nvhpc_acc) and ran into the errors shown below. For the cpu version, I was using Intel's MKL interface to fftw3 so for the code below, I compiled fftw3-3.9 for the linking process. I also added the libifcore statement you suggested to the makefile

IBIFCORE_PATH ?= /opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
LLIBS += -L$(LIBIFCORE_PATH) -lifcore


The error in this case comes from nvfortran which seems to be unhappy with missing flags. Since I have not altered any of the makefile contents nvidia tools (except to define the hpc_sdk path), this problem must lie in the original makefile. I will post the makefile.include for reference below the make output as well. I would be grateful for any further suggestions as to what to try next. (Thanks for your help!)

Code: Select all

rsync -u ../../src/*.F ../../src/*.inc .
rm -f vasp ; make vasp ; cp vasp ../../bin/vasp_std
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/std'
mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -L -lifcore -lblas -llapack -Mscalapack -L/opt/fftw3/lib -lfftw3 
nvfortran-Error-Switch -L expects an argument
-L<libdir>          Passed to linker; Add directory to library search path
make[2]: *** [vasp] Error 1
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
cp: cannot stat ‘vasp’: No such file or directory
make[1]: *** [all] Error 1
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
make: *** [std] Error 2

Here is the makefile.include file

Code: Select all

>cat makefile.include
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
              -DMPI -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dqd_emulate \
              -Dfock_dblbuf \
              -D_OPENACC \
              -DUSENCCL -DUSENCCLP2P

CPP         = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
#       to one that comes with your NVIDIA-HPC SDK
FC          = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6
FCL         = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -c++libs

FREE        = -Mfree

FFLAGS      = -Mbackslash -Mlarge_arrays

OFLAG       = -fast

DEBUG       = -Mfree -O0 -traceback

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

LLIBS       = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1  := pade_fit.o
SOURCE_O2  := pead.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = nvfortran
CC_LIB      = nvc -w
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1 -Mfixed
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT      =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC      ?= /opt/nvidia/hpc_sdk
#NVVERSION   = 21.11
#NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)

# If the above fails, then NVROOT needs to be set manually
NVHPC        = /opt/nvidia/hpc_sdk
NVVERSION   = 22.2
NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)
#

# Software emulation of quadruple precsion (mandatory)
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

IBIFCORE_PATH  ?= /opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
LLIBS       += -L$(LIBIFCORE_PATH) -lifcore


# BLAS and LAPACK (mandatory)
LLIBS      += -lblas -llapack

# scaLAPACK (mandatory)
LLIBS      += -Mscalapack

# FFTW (mandatory)
FFTW_ROOT  = /opt/fftw3
LLIBS      += -L$(FFTW_ROOT)/lib -lfftw3
INCS       += -I$(FFTW_ROOT)/include

# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT  ?= /path/to/your/hdf5/installation
#LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS    += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

[paulfons]

Here is a followup on changing the linking order. The problem seems to be associated with MKL again. The makefile.include is also included below for reference

Code: Select all

make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/std'
mpiifort -qmkl=sequential -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -Lparser -lparser -lstdc++ -Mmkl -L/opt/intel/oneapi/mkl/2022.0.1/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd 
ifort: command line warning #10006: ignoring unknown option '-Mmkl'
ld: warning: libnvhpcatm.so, needed by /opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so, not found (try using -rpath or -rpath-link)
ld: warning: libnvcpumath.so, needed by /opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so, not found (try using -rpath or -rpath-link)
ld: warning: libnvc.so, needed by /opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so, not found (try using -rpath or -rpath-link)
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_comm_null'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Comm_f2c'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_double'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmtr_end'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90_str_index_klen'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__builtin__pgi_isnandd'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_ldw_end'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `__abort_init'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__fd_exp_1'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_dblcplex'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmtw_inita'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90_str_copy_klen'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_encode_fmta'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__fd_log10_1'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_packed'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_src_info03a'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_op_max'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_op_maxloc'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_op_min'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `ftn_i_jishft'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_byte'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__fd_atan2_1'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_comm_world'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_float'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_request_null'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_cplex'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_unsigned_short'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Comm_c2f'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmt_reada'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_double_int'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_ldw_init'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_int'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_sc_ch_ldw'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_long_long_int'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_integer'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_dblprec'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmtr_init2003a'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmt_write64_aa'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Type_f2c'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Op_f2c'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `__c_mset1'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__c_mcopy1_bwd'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `__ctrl_init'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmtw_end'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_op_sum'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Op_c2f'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `__builtin__pgi_isinfdd'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `ompi_mpi_real'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90_stop08a'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqd.so: undefined reference to `__fd_log_1'
/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib/libqdmod.so: undefined reference to `pgf90io_fmt_writea'
/opt/intel/oneapi/mkl/2022.0.1/lib/intel64/libmkl_blacs_openmpi_lp64.so: undefined reference to `MPI_Type_c2f'
make[2]: *** [vasp] Error 1
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
cp: cannot stat ‘vasp’: No such file or directory
make[1]: *** [all] Error 1
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
make: *** [std] Error 2
(base) paulfons@kaon:/data/Software/Vasp/vasp.6.3.0>

makefile.include

Code: Select all

>cat makefile.include
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxIFC\" \
              -DMPI -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dfock_dblbuf

CPP         = fpp -f_com=no -free -w0  $*$(FUFFIX) $*$(SUFFIX) $(CPP_OPTIONS)

FC          = mpiifort
FCL         = mpiifort

FREE        = -free -names lowercase

FFLAGS      = -assume byterecl -w

OFLAG       = -O2
OFLAG_IN    = $(OFLAG)
DEBUG       = -O0

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = $(FC)
CC_LIB      = icc
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = icpc
LLIBS       = -lstdc++

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
FFLAGS     += -xHOST

# Intel MKL (FFTW, BLAS, LAPACK, and scaLAPACK)
# (Note: for Intel Parallel Studio's MKL use -mkl instead of -qmkl)
FCL        += -qmkl=sequential
MKLROOT    ?= /path/to/your/mkl/installation
#LLIBS      += -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
#INCS        =-I$(MKLROOT)/include/fftw
LLIBS      += -Mmkl -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
INCS       += -I$(MKLROOT)/include/fftw

NVROOT     = /opt/nvidia/hpc_sdk/Linux_x86_64/22.2
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT  = /opt/hdf5
#LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
CPP_OPTIONS    += -DVASP2WANNIER90
WANNIER90_ROOT = /opt/wannier/usr
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

[paulfons]

Another try with a fresh copy of makefile.include.nvhpc copied to makefile.include. The makefile.include is located below for reference. I added the reference to fftw3 as well as the
(missing?) lifcore library. Only one missing reference now. Suggestions?

Code: Select all

rsync -u ../../src/*.F ../../src/*.inc .
rm -f vasp ; make vasp ; cp vasp ../../bin/vasp_std
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/std'
mpif90 -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -Lparser -lparser -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -lifcore -Mscalapack -llapack -lblas -L/opt/fftw3/lib -lfftw3 
/usr/bin/ld: minimax_functions2D.o: undefined reference to symbol '__fminq'
/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin/libimf.so: error adding symbols: DSO missing from command line
make[2]: *** [vasp] Error 2
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
cp: cannot stat ‘vasp’: No such file or directory
make[1]: *** [all] Error 1
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/std'
make: *** [std] Error 2

Code: Select all

>cat makefile.include
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
              -DMPI -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dqd_emulate \
              -Dfock_dblbuf

CPP         = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)

FC          = mpif90
FCL         = mpif90 -c++libs

FREE        = -Mfree

FFLAGS      = -Mbackslash -Mlarge_arrays

OFLAG       = -fast

DEBUG       = -Mfree -O0 -traceback

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

# Redefine the standard list of O1 and O2 objects
SOURCE_O1  := pade_fit.o
SOURCE_O2  := pead.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = nvfortran
CC_LIB      = nvc -w
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1 -Mfixed
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT      =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC      ?= /opt/nvidia/hpc_sdk
#NVVERSION   = 21.11
#NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)

# Software emulation of quadruple precsion (mandatory)
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

LIBIFCORE_PATH  ?= /opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin
LLIBS       += -L$(LIBIFCORE_PATH) -lifcore


# BLAS (mandatory)
BLAS        = -lblas

# LAPACK (mandatory)
LAPACK      = -llapack

# scaLAPACK (mandatory)
SCALAPACK   = -Mscalapack

LLIBS      += $(SCALAPACK) $(LAPACK) $(BLAS)

# FFTW (mandatory)
FFTW_ROOT  ?= /opt/fftw3
LLIBS      += -L$(FFTW_ROOT)/lib -lfftw3
INCS       += -I$(FFTW_ROOT)/include

# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT  ?= /path/to/your/hdf5/installation
#LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS    += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

[marie-therese.huebsch]

In the first trial for the makefile.include.nvhpc_acc, you have a typo in the libifcore statement. In the first line LIBIFCORE_PATH should be defined and then it is used as an argument for the -L compiler option. Because IBIFCORE_PATH instead of LIBIFCORE_PATH is defined you get the error

Code: Select all

nvfortran-Error-Switch -L expects an argument

In the last try with a fresh copy of makefile.include.nvhpc, you did not repeat the same typo for LIBIFCORE_PATH. However, you have the same issue as before when using makefile.include.nvhpc_ompi_mkl_omp_acc with libifcore added.

Code: Select all

/usr/bin/ld: minimax_functions2D.o: undefined reference to symbol '__fminq'
/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin/libimf.so: error adding symbols: DSO missing from command line

Let us focus on makefile.include.nvhpc_acc. Please start a fresh installation (make veryclean) and try compiling without the libifcore statement. What do you get then?

[paulfons]

As suggested, I focused on makefile.include.nvhpc_acc and added the ifcore library as well as my locally compiled fftw3 libraries. After making very clean, I ran make and it ran to completion without errors. In the bin directory, I note the usual three binaries vasp_gam, vasp_std, and vasp_ncl. When I tried to run them, I encountered a run time error with the quad precision library even though there were no linking errors in the make output (the vasp_gam output is pasted below). It seems like progress, but I don't understand where the vasp_gpu binary is and where the runtime error is coming from. Thanks for any suggestions!

Code: Select all

/data/Software/Vasp/vasp.6.3.0/bin/vasp_std: error while loading shared libraries: libqdmod.so.0: cannot open shared object file: No such file or directory
/data/Software/Vasp/vasp.6.3.0/bin/vasp_std: error while loading shared libraries: libqdmod.so.0: cannot open shared object file: No such file or directory

Code: Select all

make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/gam'
mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -lblas -llapack -Mscalapack -L/opt/fftw3/lib -lfftw3 -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -lifcore 
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/gam'
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/gam'
if [ ! -d build/ncl ] ; then mkdir -p build/ncl  ; fi
cp src/makefile src/.objects src/makedeps.awk makefile.include build/ncl 
make -C build/ncl VERSION=ncl check
make[1]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
exit 0
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
make -C build/ncl VERSION=ncl cleandependencies -j1
make[1]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
rm -f .depend
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
make -C build/ncl VERSION=ncl all
make[1]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
rsync -ru ../../src/lib .
cp makefile.include lib
make -C lib -j1
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/lib'
make libdmy.a
make[3]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/lib'
make[3]: `libdmy.a' is up to date.
make[3]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/lib'
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/lib'
rsync -ru ../../src/parser .
cp makefile.include parser
make -C parser -j1
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/parser'
make libparser.a
make[3]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/parser'
make[3]: `libparser.a' is up to date.
make[3]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/parser'
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl/parser'
rsync -u ../../src/*.F ../../src/*.inc .
rm -f vasp ; make vasp ; cp vasp ../../bin/vasp_ncl
make[2]: Entering directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.6 -c++libs -o vasp c2f_interface.o nccl2for.o simd.o base.o profiling.o string.o tutor.o version.o command_line.o vhdf5_base.o incar_reader.o reader_base.o openmp.o openacc_struct.o mpi.o mpi_shmem.o mathtools.o hamil_struct.o radial_struct.o pseudo_struct.o mgrid_struct.o wave_struct.o nl_struct.o mkpoints_struct.o poscar_struct.o afqmc_struct.o fock_glb.o chi_glb.o smart_allocate.o xml.o extpot_glb.o constant.o ml_ff_c2f_interface.o ml_ff_prec.o ml_ff_constant.o ml_ff_taglist.o ml_ff_struct.o ml_ff_mpi_help.o ml_ff_mpi_shmem.o vdwforcefield_glb.o jacobi.o main_mpi.o openacc.o scala.o asa.o lattice.o poscar.o ini.o mgrid.o ml_ff_error.o ml_ff_mpi.o ml_ff_helper.o ml_ff_logfile.o ml_ff_math.o ml_ff_iohandle.o ml_ff_memory.o ml_ff_abinitio.o ml_ff_ff.o ml_ff_mlff.o setex_struct.o xclib.o vdw_nl.o xclib_grad.o setex.o radial.o pseudo.o gridq.o ebs.o symlib.o mkpoints.o random.o wave.o wave_mpi.o wave_high.o bext.o spinsym.o symmetry.o lattlib.o nonl.o nonlr.o nonl_high.o dfast.o choleski2.o mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o constrmag.o cl_shift.o relativistic.o LDApU.o paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o diis.o rhfatm.o hyperfine.o fock_ace.o paw.o mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o solvation.o scpc.o pot.o tet.o dos.o elf.o hamil_rot.o chain.o dyna.o fileio.o vhdf5.o sphpro.o us.o core_rel.o aedens.o wavpre.o wavpre_noio.o broyden.o dynbr.o reader.o writer.o xml_writer.o brent.o stufak.o opergrid.o stepver.o chgloc.o fast_aug.o fock_multipole.o fock.o fock_dbl.o fock_frc.o mkpoints_change.o subrot_cluster.o sym_grad.o mymath.o npt_dynamics.o subdftd3.o subdftd4.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o nmr.o pead.o k-proj.o subrot.o subrot_scf.o paircorrection.o rpa_force.o ml_interface.o force.o pwlhf.o gw_model.o optreal.o steep.o rmm-diis.o davidson.o david_inner.o root_find.o lcao_bare.o locproj.o electron_common.o electron.o rot.o electron_all.o shm.o pardens.o optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o hamil_lr.o rmm-diis_lr.o subrot_lr.o lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o linear_optics.o setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o gauss_quad.o m_unirnk.o minimax_ini.o minimax_dependence.o minimax_functions1D.o minimax_functions2D.o minimax_struct.o minimax_varpro.o minimax.o umco.o mlwf.o ratpol.o pade_fit.o screened_2e.o wave_cacher.o crpa.o chi_base.o wpot.o local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o bse_te.o bse.o time_propagation.o acfdt.o afqmc.o rpax.o chi.o acfdt_GG.o dmft.o GG_base.o greens_orbital.o lt_mp2.o rnd_orb_mp2.o greens_real_space.o chi_GG.o chi_super.o sydmat.o rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o wave_interpolate.o linear_response.o auger.o dmatrix.o phonon.o wannier_mats.o elphon.o core_con_mat.o embed.o extpot.o rpa_high.o fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o main.o  -Llib -ldmy -Lparser -lparser -cudalib=cublas,cusolver,cufft,nccl -cuda -L/opt/nvidia/hpc_sdk/Linux_x86_64/22.2/compilers/extras/qd/lib -lqdmod -lqd -lblas -llapack -Mscalapack -L/opt/fftw3/lib -lfftw3 -L/opt/intel/oneapi/compiler/2022.0.1/linux/compiler/lib/intel64_lin -lifcore 
make[2]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'
make[1]: Leaving directory `/data/Software/Vasp/vasp.6.3.0/build/ncl'

[marie-therese.huebsch]

I think that is progress! When compiling VASP to run on GPUs, you will obtain the usual vasp_gam, vasp_std, and vasp_ncl. The part that is ported to GPUs (which is the computationally heavy part) will then automatically run on GPUs. Did you see any mention of vasp_gpu on the Wiki, or was that from experience? (In a previous attempt to port VASP to GPUs, we introduced a new executable vasp_gpu, but this is deprecated.)

Regarding the error at runtime. It looks like the library is not in your system library path. Can you try to do the following and rerun the test calculation:

Code: Select all

export QD_LIB = your/nvidia/hpc_sdk/installation/compilers/extras/qd/lib
export LD_LIBRARY_PATH=$QD_LIB:$LD_LIBRARY_PATH

If that works, I'd suggest adding this to your bashrc and submission scripts.

[paulfons]

Thank you for the suggestion and sorry to be late in getting back with a result. I tried running the new successfully linked binary vasp_ncl to run a simple GaAs calculation and encountered the following error. Any suggestions as to what to try next? I do know that the vendor that installed the new node with the Ampere card ran the cuda many body sample program without problem so the card is functional. I also ran a few of the cuda-samples successfully using the included makefile and nvfortran.

The .bashrc setup includes the following lines

Code: Select all

	NVARCH=`uname -s`_`uname -m`; export NVARCH
	NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS
	MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/22.2/compilers/man; export MANPATH
	PATH=$NVCOMPILERS/$NVARCH/22.2/compilers/bin:$PATH; export PATH
	export PATH=$NVCOMPILERS/$NVARCH/22.2/comm_libs/mpi/bin:$PATH
	export MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/22.2/comm_libs/mpi/man

Here is the attempt to run an existing (works fine with the non gpu vasp_ncl) sample calculation on GaAs

Code: Select all

(base) paulfons@kaon:/data/Vasp/GaAs>/data/Software/Vasp/vasp.6.3.0/bin/vasp_ncl
[kaon:13793] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ../../../../../orte/mca/ess/singleton/ess_singleton_module.c at line 388
[kaon:13793] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ../../../../../orte/mca/ess/singleton/ess_singleton_module.c at line 166
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orte_init:startup:internal-failure
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/22.2/openmpi/209518-rel-1/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:startup:internal-failure
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/22.2/openmpi/209518-rel-1/share/openmpi/help-mpi-runtime.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[kaon:13793] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!