problems with fftw (5.2.8 and 5.2.11)

[fabio.chiatti]

Hi everyone,
I've got a problem when trying to compile VASP5.2 (either 5.2.8 or 5.2.11) in parallel. When I use the Juergen Furthmueller, everything is ok but I noticed that, when using this fft, VASP5.2 takes more time to calculate than VASP4.6 (compiled with the fftw).
When I compile with the fftw, this is the error printed:


/opt/Openmpi-ifort_11.1/bin/mpif77 -FR -lowercase -assume byterecl -O2 -c fftmpiw.f90
fftmpiw.f90(59): error #5102: Cannot open include file 'fftw3.f'
include 'fftw3.f'
--------------^
fftmpiw.f90(101): error #5102: Cannot open include file 'fftw3.f'
include 'fftw3.f'
--------------^
fftmpiw.f90(122): error #5102: Cannot open include file 'fftw3.f'
include 'fftw3.f'
--------------^
fftmpiw.f90(237): error #5102: Cannot open include file 'fftw3.f'
include 'fftw3.f'
--------------^
compilation aborted for fftmpiw.f90 (code 1)
make: *** [fftmpiw.o] Error 1





Here is the Makefile:

.SUFFIXES: .inc .f .f90 .F
#-----------------------------------------------------------------------
# Makefile for Intel Fortran compiler for Pentium/Athlon/Opteron
# bases systems
# we recommend this makefile for both Intel as well as AMD systems
# for AMD based systems appropriate BLAS and fftw libraries are
# however mandatory (whereas they are optional for Intel platforms)
#
# The makefile was tested only under Linux on Intel and AMD platforms
# the following compiler versions have been tested:
# - ifc.7.1 works stable somewhat slow but reliably
# - ifc.8.1 fails to compile the code properly
# - ifc.9.1 recommended (both for 32 and 64 bit)
# - ifc.10.1 partially recommended (both for 32 and 64 bit)
# tested build 20080312 Package ID: l_fc_p_10.1.015
# the gamma only mpi version can not be compiles
# using ifc.10.1
#
# it might be required to change some of library pathes, since
# LINUX installation vary a lot
# Hence check ***ALL*** options in this makefile very carefully
#-----------------------------------------------------------------------
#
# BLAS must be installed on the machine
# there are several options:
# 1) very slow but works:
# retrieve the lapackage from ftp.netlib.org
# and compile the blas routines (BLAS/SRC directory)
# please use g77 or f77 for the compilation. When I tried to
# use pgf77 or pgf90 for BLAS, VASP hang up when calling
# ZHEEV (however this was with lapack 1.1 now I use lapack 2.0)
# 2) more desirable: get an optimized BLAS
#
# the two most reliable packages around are presently:
# 2a) Intels own optimised BLAS (PIII, P4, PD, PC2, Itanium)
# http://developer.intel.com/software/products/mkl/
# this is really excellent, if you use Intel CPU's
#
# 2b) probably fastest SSE2 (4 GFlops on P4, 2.53 GHz, 16 GFlops PD,
# around 30 GFlops on Quad core)
# Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
# http://www.tacc.utexas.edu/resources/software/
#
#-----------------------------------------------------------------------

# all CPP processed fortran files have the extension .f90
SUFFIX=.f90

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifc
# fortran linker
FCL=$(FC)


#-----------------------------------------------------------------------
# whereis CPP ?? (I need CPP, can't use gcc with proper options)
# that's the location of gcc for SUSE 5.3
#
# CPP_ = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C
#
# that's probably the right line for some Red Hat distribution:
#
# CPP_ = /usr/lib/gcc-lib/i386-redhat-linux/2.7.2.3/cpp -P -C
#
# SUSE X.X, maybe some Red Hat distributions:

CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)

#-----------------------------------------------------------------------
# possible options for CPP:
# NGXhalf charge density reduced in X direction
# wNGXhalf gamma point only reduced in X direction
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------

CPP = $(CPP_) -DHOST=\"LinuxIFC\" \
-Dkind8 -DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGXhalf \
# -DRPROMU_DGEMV -DRACCMU_DGEMV

#-----------------------------------------------------------------------
# general fortran flags (there must a trailing blank on this line)
# byterecl is strictly required for ifc, since otherwise
# the WAVECAR file becomes huge
#-----------------------------------------------------------------------

FFLAGS = -FR -lowercase -assume byterecl

#-----------------------------------------------------------------------
# optimization
# we have tested whether higher optimisation improves performance
# -axK SSE1 optimization, but also generate code executable on all mach.
# xK improves performance somewhat on XP, and a is required in order
# to run the code on older Athlons as well
# -xW SSE2 optimization
# -axW SSE2 optimization, but also generate code executable on all mach.
# -tpp6 P3 optimization
# -tpp7 P4 optimization
#-----------------------------------------------------------------------

# ifc.9.1, ifc.10.1 recommended
OFLAG=-O2

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)

#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# VASP works fastest with the libgoto library
# so that's what we recommend
#-----------------------------------------------------------------------

# mkl.10.0
# set -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines
#BLAS=-L/opt/intel/mkl100/lib/em64t -lmkl -lpthread

# even faster for VASP Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
# parallel goto version requires sometimes -libverbs
#BLAS= /opt/libs/libgoto/libgoto.so

# LAPACK, simplest use vasp.5.lib/lapack_double
#LAPACK= ../vasp.5.lib/lapack_double.o

# use the mkl Intel lapack
#LAPACK= -lmkl_lapack

MKLPATH=/opt/intel/mkl/10.0.3.020/lib/em64t
BLAS=$(MKLPATH)/libmkl_solver_lp64_sequential.a $(MKLPATH)/dlasrt2.f -Wl,--start-group $(MKLPATH)/libmkl_intel_lp64.a $(MKLPATH)/libmkl_sequential.a $(MKLPATH)/libmkl_core.a -Wl,--end-group -lpthread

#-----------------------------------------------------------------------

LIB = -L../vasp.5.lib -ldmy \
../vasp.5.lib/linpack_double.o $(LAPACK) \
$(BLAS)

# options for linking, nothing is required (usually)
LINK =

#-----------------------------------------------------------------------
# fft libraries:
# VASP.5.2 can use fftw.3.1.X (http://www.fftw.org)
# since this version is faster on P4 machines, we recommend to use it
#-----------------------------------------------------------------------

FFT3D = fft3dfurth.o fft3dlib.o

# alternatively: fftw.3.1.X is slighly faster and should be used if available
#FFT3D = fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a


#=======================================================================
# MPI section, uncomment the following lines until
# general rules and compile lines
# presently we recommend OPENMPI, since it seems to offer better
# performance than lam or mpich
#
# !!! Please do not send me any queries on how to install MPI, I will
# certainly not answer them !!!!
#=======================================================================
#-----------------------------------------------------------------------
# fortran linker for mpi
#-----------------------------------------------------------------------

FC=/opt/Openmpi-ifort_11.1/bin/mpif77
#FCL=$(FC)

#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------

#-----------------------------------------------------------------------

CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
-Dkind8 -DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
-DMPI_BLOCK=8000 -DRPROMU_DGEMV -DRACCMU_DGEMV



#-----------------------------------------------------------------------
# location of SCALAPACK
# if you do not use SCALAPACK simply leave that section commented out
#-----------------------------------------------------------------------

#BLACS=$(HOME)/archives/SCALAPACK/BLACS/
#SCA_=$(HOME)/archives/SCALAPACK/SCALAPACK

#SCA= $(SCA_)/libscalapack.a \
# $(BLACS)/LIB/blacsF77init_MPI-LINUX-0.a $(BLACS)/LIB/blacs_MPI-LINUX-0.a $(BLACS)/LIB/blacsF77init_MPI-LINUX-0.a

SCA=

#-----------------------------------------------------------------------
# libraries for mpi
#-----------------------------------------------------------------------

LIB = -L../vasp.5.lib -ldmy \
../vasp.5.lib/linpack_double.o $(LAPACK) \
$(SCA) $(BLAS)

# FFT: fftmpi.o with fft3dlib of Juergen Furthmueller
#FFT3D = fftmpi.o fftmpi_map.o fft3dfurth.o fft3dlib.o

# alternatively: fftw.3.1.X is slighly faster and should be used if available
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/libfftw3xf_intel.a

#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------
BASIC= symmetry.o symlib.o lattlib.o random.o


SOURCE= base.o mpi.o smart_allocate.o xml.o \
constant.o jacobi.o main_mpi.o scala.o \
asa.o lattice.o poscar.o ini.o xclib.o xclib_grad.o \
radial.o pseudo.o mgrid.o gridq.o ebs.o \
mkpoints.o wave.o wave_mpi.o wave_high.o \
$(BASIC) nonl.o nonlr.o nonl_high.o dfast.o choleski2.o \
mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o \
constrmag.o cl_shift.o relativistic.o LDApU.o \
paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o rhfatm.o paw.o \
mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o pot.o \
dos.o elf.o tet.o tetweight.o hamil_rot.o \
steep.o chain.o dyna.o sphpro.o us.o core_rel.o \
aedens.o wavpre.o wavpre_noio.o broyden.o \
dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o \
brent.o stufak.o fileio.o opergrid.o stepver.o \
chgloc.o fast_aug.o fock.o mkpoints_change.o sym_grad.o \
mymath.o optengines.o internals.o hessian.o gadget.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o \
hamil_high.o nmr.o force.o \
pead.o mlwf.o subrot.o subrot_scf.o pwlhf.o gw_model.o optreal.o davidson.o david_inner.o \
electron.o rot.o electron_all.o shm.o pardens.o paircorrection.o \
optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o \
hamil_lr.o rmm-diis_lr.o subrot_cluster.o subrot_lr.o \
lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o \
linear_optics.o linear_response.o \
setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o \
ratpol.o screened_2e.o wave_cacher.o chi_base.o wpot.o local_field.o \
ump2.o bse.o acfdt.o chi.o sydmat.o

vasp: $(SOURCE) $(FFT3D) $(INC) main.o
rm -f vasp
$(FCL) -o vasp main.o $(SOURCE) $(FFT3D) $(LIB) $(LINK)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F $(INC)
$(FCL) -o makeparam $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o $(INC)
$(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o $(INC)
$(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D) $(INC)
$(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $(INC)
$(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)

clean:
-rm -f *.g *.f *.o *.L *.mod ; touch *.F

main.o: main$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcspin$(SUFFIX)

makeparam.o: makeparam$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c makeparam$(SUFFIX)

makeparam$(SUFFIX): makeparam.F main.F
#
# MIND: I do not have a full dependency list for the include
# and MODULES: here are only the minimal basic dependencies
# if one strucuture is changed then touch_dep must be called
# with the corresponding name of the structure
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
poscar.o: poscar.inc poscar.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F

$(OBJ_HIGH):
$(CPP)
$(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
$(CPP)
$(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)

fft3dlib_f77.o: fft3dlib_f77.F
$(CPP)
$(F77) $(FFLAGS_F77) -c $*$(SUFFIX)

.F.o:
$(CPP)
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
$(CPP)
$(SUFFIX).o:
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)

# special rules
#-----------------------------------------------------------------------
# these special rules are cummulative (that is once failed
# in one compiler version, stays in the list forever)
# -tpp5|6|7 P, PII-PIII, PIV
# -xW use SIMD (does not pay of on PII, since fft3d uses double prec)
# all other options do no affect the code performance since -O1 is used

fft3dlib.o : fft3dlib.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)

fft3dfurth.o : fft3dfurth.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

fftw3d.o : fftw3d.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

wave_high.o : wave_high.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

radial.o : radial.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

symlib.o : symlib.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

symmetry.o : symmetry.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

wave_mpi.o : wave_mpi.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

wave.o : wave.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

dynbr.o : dynbr.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

asa.o : asa.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

broyden.o : broyden.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)

us.o : us.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)

LDApU.o : LDApU.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)

Thanks in advance!
Fabio

[alex]

Hello Fabio,

the compiler searches the included file fftw3.h and can't find it. So go to your fftw directory, subdirectory include. There will be the file. Copy to your vasp-dir.
You can also compile with -I$PATH_TO_FFTW. Should also work.

Cheers,

alex

[fabio.chiatti]

Thank you very much, Alex!
That problem has been solved, but a new one has occured.
I can't understand what is going on...
Thank you very much in advance...

/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/libfftw3xf_intel.a(fftw_plan_guru64_dft.o): In function `fftw_plan_guru64_dft':
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft.c:(.text+0xbd): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft.c:(.text+0x520): undefined reference to `DftiCreateDescriptor_d_1d'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft.c:(.text+0x556): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft.c:(.text+0x57f): undefined reference to `DftiCreateDescriptor_d_1d'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/libfftw3xf_intel.a(fftw_plan_guru64_dft_c2r.o): In function `fftw_plan_guru64_dft_c2r':
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_c2r.c:(.text+0xae): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_c2r.c:(.text+0x331): undefined reference to `DftiCreateDescriptor_d_1d'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_c2r.c:(.text+0x354): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_c2r.c:(.text+0x36d): undefined reference to `DftiCreateDescriptor_d_1d'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/libfftw3xf_intel.a(fftw_plan_guru64_dft_r2c.o): In function `fftw_plan_guru64_dft_r2c':
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_r2c.c:(.text+0xae): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_r2c.c:(.text+0x331): undefined reference to `DftiCreateDescriptor_d_1d'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_r2c.c:(.text+0x354): undefined reference to `DftiCreateDescriptor_d_md'
/opt/intel/composerxe-2011.0.084/mkl/interfaces/fftw3xf/wrappers/fftw_plan_guru64_dft_r2c.c:(.text+0x36d): undefined reference to `DftiCreateDescriptor_d_1d'

[alex]

Why are you using an MKL interface???

[fabio.chiatti]

I cannot answer this technical question as the fft library was not compiled by me. What I know is that the previous VASP4.6 Makefile could successfully be compiled using the same library and now I can't anymore... why?

[alex]

Hello Fabio,

why? Things change. From 4.6.34 to 5.2.11 there are years and years in development. Don't worry too much about the past and try not to figure out why it's not working anymore. Learn from that and look forward. You are going to learn more!

Question, before we start digging deep: Are you using Intel or AMD CPUs?

Background: if you are using AMD it's probably not the best idea to link versus MKL ...

Cheers,

alex

[fabio.chiatti]

I'm sorry I didn't tell you before: it's an INTEL CPU :-)

<span class='smallblacktext'>[ Edited Thu Dec 09 2010, 05:13PM ]</span>

[alex]

oki, please check if it links to Furthmüllers fft?

[fabio.chiatti]

When linking to the furthmuller everything is ok. I can compile VASP and make calculations (that's actually what I'm doing right now), but it is much slower than the previous 4.6 (as I'm not using the fftw).

[alex]

What happens if you link without the mkl-fftw3 wrapper? I don't think you'll actually need it.

DftiCreateDescriptor_d_1d seems to be an internal mkl interface. It might be not defined in your wrapper library. the file with the definition is mkl_dfti.h in your mkl distribution.

Cheers,

alex