Installing VASP CPU only version on AMD EPYC 7742

[david_keller]

Which would be the appropriate starting makefile to use to compile 6.2 on nodes comprised of AMD EPYC 7742 64-Core Processors, running 5.3.0-62-generic #56~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux? The CPUs are AMD EPYC 7742 64-Core Processors. I want to compile a CPU only version and an OpenACC version.

[andreas.singraber]

Hi!

I am working on a Linux Mint 20.1 machine with AMD Epyc 7402P processor (24 cores), so it looks like a very similar setup:

Code: Select all

Linux 5.4.0-70-generic #78-Ubuntu SMP Fri Mar 19 13:29:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I have not yet tried anything special but starting from the makefile.include

Code: Select all

arch/makefile.include.linux_gnu_omp

I could compile the CPU version with gfortran 9.3 and the system libraries. You may need to install some packages:

Code: Select all

sudo apt install libscalapack-openmpi-dev libfftw3-dev libopenblas-dev

With this setup the "fast" testsuite runs without errors. Here is the makefile.include I used:

Code: Select all

# Precompiler options
CPP_OPTIONS= -DHOST=\"LinuxGNU\" \
             -DMPI -DMPI_BLOCK=8000 -Duse_collective \
             -DscaLAPACK \
             -DCACHE_SIZE=4000 \
             -Davoidalloc \
             -Dvasp6 \
             -Duse_bse_te \
             -Dtbdyn \
             -Dfock_dblbuf \
             -D_OPENMP

CPP        = gcc -E -P -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS)

FC         = mpif90 -fopenmp
FCL        = mpif90 -fopenmp

FREE       = -ffree-form -ffree-line-length-none

FFLAGS     = -w -march=native
OFLAG      = -O2
OFLAG_IN   = $(OFLAG)
DEBUG      = -O0

BLAS       = -lopenblas
LAPACK     =
BLACS      = 
SCALAPACK  = -lscalapack-openmpi $(BLACS)

LLIBS      = $(SCALAPACK) $(LAPACK) $(BLAS)

FFTW       ?= 
LLIBS      += -lfftw3 -lfftw3_omp
INCS       = -I/usr/include

OBJECTS    = fftmpiw.o fftmpi_map.o  fftw3d.o  fft3dlib.o

OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o

# For what used to be vasp.5.lib
CPP_LIB    = $(CPP)
FC_LIB     = $(FC)
CC_LIB     = gcc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB   = $(FREE)

OBJECTS_LIB= linpack_double.o getshmem.o

# For the parser library
CXX_PARS   = g++
LLIBS      += -lstdc++

### For the fft library
##CXX_FFTLIB = g++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE 
##INCS_FFTLIB= -I./include -I$(FFTW)/include
##LIBS       += fftlib
##LLIBS      += -ldl

# Normally no need to change this
SRCDIR     = ../../src
BINDIR     = ../../bin

I am currently looking into the AMD Optimizing Compiler (AOCC, https://developer.amd.com/amd-aocc/) and Libraries (AOCL, https://developer.amd.com/amd-aocl/) but I cannot give you a recommendation yet.

Please also have a look at the Wiki page regarding the installation of VASP (in particular for the OpenACC version): wiki/index.php/Installing_VASP.6.X.X

[tromanfl]

Thank you @andreas.singraber for the detailed reply. I was wondering if you / the VASP team have/s made progress on a makefile for AMD Optimizing Compiler 3.1 and AMD Optimizing Libraries 3.0? Like @david_keller who started this thread, I also need to optimize VASP for the AMD EPYC family of processors (including 7742). Thank you so much!

[andreas.singraber]

Hi!

Unfortunately I didn't find the time to do this yet.. however, it is on my table and getting a more pressing issue. I cannot promise a date for a satisfying answer but I will be working on the performance tests in the coming days and weeks. I'm sorry I have no answer to your questions at this point..

Best,
Andreas Singraber

[tromanfl]

Thanks a lot for committing to this as well Andreas! There seems to be a growing group that needs a solid solution for VASP on AMD EPYC. I am trying out several possible solutions myself, but I have not been successful to date. Will post my solution here when I find it!

[andreas.singraber]

Hello again!

First of all, I'm sorry for the long delay... it turned out to be quite tricky to get good performance on these machines but I can report that in the meantime we already found a working setup! Unfortunately, we had to alter some of our internal OpenMP usage and therefore a simple solution with only a new makefile.include will not be sufficient. However, the source code changes and a recommended makefile.include for the AMD EPYC architecture will be delivered with the upcoming VASP release (in the very near future)... please stay tuned!!

Best,

Andreas Singraber

[shaldar]

Thanks Andreas. We are looking forward to this new release with support for AMD EPYC 7742.

[davidnormal]

Hi,

I also managed to compile the vasp6.2.1 with AOCC clang and tried to link them to the AOCL libraries with every optimisation flag I know. Unfortunately it is just slower than using the Intel compilers... Maybe just looking forward to the next version