vasp.5.2 with openmpi thinks it's running on only one core

[cpp6f]

I've compiled parallel vasp 5.2 using:
intel/11.1.046
mkl
openmpi/1.3.3
fftw/3.2.2

When I run it on an 8 core node, it runs 8 independent copies of vasp that each thinks it's running on only one core:

running on 1 nodes
distr: one band on 1 nodes, 1 groups

The output is similar to the output in this thread:

http://cms.mpi.univie.ac.at/vasp-forum/ ... hp?2.14853

b/c each node thinks it's rank 0 so they all write the same output to the same file but I certainly compiled the parallel version

Normally, I would assume it's a problem with the mpi installation, the runtiime environment, or the mpiexec command:

mpiexec -x LD_LIBRARY_PATH -np 8 vasp5

But I use the exact same command to run an identically compiled vasp 4.6 and it works fine. The two versions of vasp are compiled using the same makefile except for the SOURCE variable (see below). The even weirder thing is that I got vasp 5.2 to work before on this cluster but I've lost the makefile that I used. That version ran fine using the same mpi installation, runtime environment, and mpiexec command I'm using for the new version. The problem seems to be that the following line in mpi.F is returning a rank of 0 for all of the 8 nodes instead of 0, 1, 2, 3, ...


call MPI_comm_rank( COMM%MPI_COMM, COMM%NODE_ME, ierror)


any ideas?







Here is the makefile for vasp 5.2:


.SUFFIXES: .inc .f .f90 .F


# all CPP processed fortran files have the extension .f90
SUFFIX=.f90

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------

FCL=$(FC)

FFLAGS = $(FREE) -assume byterecl

#-----------------------------------------------------------------------
# optimization
#-----------------------------------------------------------------------

OFLAG = -O2

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =

DEBUG = -g -O0
INLINE = $(OFLAG)


#------------------------------------------------------------------------------------
# whereis CPP ?? (I need CPP, can't use gcc with proper options)
#------------------------------------------------------------------------------------

CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)
#CPP_ = fpp -f_com=no -free -w0 $*.F $*$(SUFFIX)

#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------

CPP = $(CPP_) -DMPI -DHOST=\"Atlas\" -DNGZhalf -DusePSCENC -Davoidalloc -DMPI_BLOCK=8000 -DIFC -DPGF90 -Duse_collective



#-----------------------------------------------------------------------
# libraries
#-----------------------------------------------------------------------

FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
FFT3DLIB = $(FFTW_LIB) $(FFTW_LINK)
FFT3DINC = $(FFTW_INC)

BLAS =
LAPACK = $(MATH_LIB) $(MATH_LINK)
SCA =

LINPACK = ../vasp.5.lib/linpack_double.o
VASPLIB = ../vasp.5.lib/libdmy.a

LIB = $(BLAS) $(LAPACK) $(SCA) $(FFT3DLIB) $(LINPACK) $(VASPLIB)

INC = $(FFT3DINC)


#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------
BASIC= symmetry.o symlib.o lattlib.o random.o

VTST= dimer.o dynmat.o neb.o lanczos.o sd.o cg.o qm.o lbfgs.o bfgs.o fire.o opt.o

EXT= quambo.o netcdf.o

SOURCE= base.o mpi.o smart_allocate.o xml.o \
constant.o jacobi.o main_mpi.o scala.o \
asa.o lattice.o poscar.o ini.o mgrid.o xclib.o vdw_nl.o xclib_grad.o \
radial.o pseudo.o gridq.o ebs.o \
mkpoints.o wave.o wave_mpi.o wave_high.o \
$(BASIC) nonl.o nonlr.o nonl_high.o dfast.o choleski2.o \
mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o \
constrmag.o cl_shift.o relativistic.o LDApU.o \
paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o rhfatm.o paw.o \
mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o pot.o \
dos.o elf.o tet.o tetweight.o hamil_rot.o \
steep.o $(VTST) chain.o dyna.o sphpro.o us.o core_rel.o \
aedens.o wavpre.o wavpre_noio.o broyden.o \
dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o \
brent.o stufak.o fileio.o opergrid.o stepver.o \
chgloc.o fast_aug.o fock.o mkpoints_change.o sym_grad.o \
mymath.o internals.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o \
hamil_high.o nmr.o pead.o mlwf.o subrot.o subrot_scf.o \
force.o pwlhf.o gw_model.o optreal.o davidson.o david_inner.o \
electron.o rot.o electron_all.o shm.o pardens.o paircorrection.o \
optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o \
hamil_lr.o rmm-diis_lr.o subrot_cluster.o subrot_lr.o \
lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o \
linear_optics.o linear_response.o \
setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o \
ratpol.o screened_2e.o wave_cacher.o chi_base.o wpot.o local_field.o \
ump2.o bse_te.o bse.o acfdt.o chi.o sydmat.o dmft.o \
rmm-diis_mlr.o linear_response_NMR.o $(EXT)



vasp: $(SOURCE) $(FFT3D) main.o
rm -f vasp
$(FCL) -o vasp main.o $(SOURCE) $(FFT3D) $(LIB) $(LINK)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F
$(FCL) -o makeparam $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o
$(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o
$(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D)
$(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $
$(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)




clean:
-rm -f *.g *.f *.f90 *.o *.L *.mod ; touch *.F

main.o: main$(SUFFIX)
$(FC) $(FFLAGS) $(DEBUG) $(INC) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INC) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INC) -c xcspin$(SUFFIX)

makeparam.o: makeparam$(SUFFIX)
$(FC) $(FFLAGS) $(DEBUG) $(INC) -c makeparam$(SUFFIX)

makeparam$(SUFFIX): makeparam.F main.F



#
# MIND: I do not have a full dependency list for the include
# and MODULES: here are only the minimal basic dependencies
# if one strucuture is changed then touch_dep must be called
# with the corresponding name of the structure
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
poscar.o: poscar.inc poscar.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.inc wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F

$(OBJ_HIGH):
$(CPP)
$(FC) $(FFLAGS) $(OFLAG_HIGH) $(INC) -c $*$(SUFFIX)
$(OBJ_NOOPT):
$(CPP)
$(FC) $(FFLAGS) $(INC) -c $*$(SUFFIX)

fft3dlib_f77.o: fft3dlib_f77.F
$(CPP)
$(F77) $(FFLAGS_F77) -c $*$(SUFFIX)

.F.o:
$(CPP)
$(FC) $(FFLAGS) $(OFLAG) $(INC) -c $*$(SUFFIX)
.F$(SUFFIX):
$(CPP)
$(SUFFIX).o:
$(FC) $(FFLAGS) $(OFLAG) $(INC) -c $*$(SUFFIX)








And for vasp 4.6


.SUFFIXES: .inc .f .f90 .F


# all CPP processed fortran files have the extension .f90
SUFFIX=.f90

#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------

FCL=$(FC)

FFLAGS = $(FREE) -assume byterecl

#-----------------------------------------------------------------------
# optimization
#-----------------------------------------------------------------------

OFLAG = -O2

OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =

DEBUG = -g -O0
INLINE = $(OFLAG)


#------------------------------------------------------------------------------------
# whereis CPP ?? (I need CPP, can't use gcc with proper options)
#------------------------------------------------------------------------------------

CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)

#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------

CPP = $(CPP_) -DMPI -DHOST=\"Atlas\" -DNGZhalf -DusePSCENC -Davoidalloc -DMPI_BLOCK=8000 -DIFC -Duse_collective -DDISP



#-----------------------------------------------------------------------
# libraries
#-----------------------------------------------------------------------

FFT3D = fftmpiw.o fftmpi_map.o fft3dlib.o
FFT3DLIB = $(FFTW_LIB) $(FFTW_LINK)
FFT3DINC = $(FFTW_INC)

BLAS =
LAPACK = $(MATH_LIB) $(MATH_LINK)
SCA =

LINPACK = ../vasp.4.lib/linpack_double.o
VASPLIB = ../vasp.4.lib/libdmy.a

LIB = $(BLAS) $(LAPACK) $(SCA) $(FFT3DLIB) $(VASPLIB)

INCS = $(FFT3DINC)


#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------

SOURCE= base.o mpi.o smart_allocate.o xml.o constant.o jacobi.o main_mpi.o scala.o asa.o \
lattice.o poscar.o ini.o setex.o radial.o pseudo.o mgrid.o mkpoints.o wave.o \
wave_mpi.o symmetry.o symlib.o lattlib.o random.o nonl.o nonlr.o dfast.o \
choleski2.o mix.o charge.o xcgrad.o xcspin.o potex1.o potex2.o metagga.o \
constrmag.o pot.o cl_shift.o force.o dos.o elf.o tet.o hamil.o steep.o dimer.o \
dynmat.o neb.o lanczos.o sd.o cg.o qm.o lbfgs.o bfgs.o fire.o opt.o chain.o \
dyna.o relativistic.o LDApU.o sphpro.o paw.o us.o ebs.o wavpre.o wavpre_noio.o \
broyden.o dynbr.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o brent.o \
stufak.o fileio.o opergrid.o stepver.o dipol.o xclib.o chgloc.o subrot.o \
optreal.o davidson.o edtest.o electron.o shm.o pardens.o paircorrection.o \
optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o setlocalpp.o aedens.o \
quambo.o netcdf.o disp.o dispersion.o


vasp: $(SOURCE) $(FFT3D) main.o
rm -f vasp
$(FCL) -o vasp $(LINK) main.o $(SOURCE) $(FFT3D) $(LINPACK) $(LIB)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F
$(FCL) -o makeparam $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o
$(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o
$(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D)
$(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $
$(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)

clean:
-rm -f *.g *.f *.f90 *.o *.L *.mod ; touch *.F

main.o: main$(SUFFIX)
$(FC) $(FFLAGS) $(DEBUG) $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcspin$(SUFFIX)

makeparam.o: makeparam$(SUFFIX)
$(FC) $(FFLAGS) $(DEBUG) $(INCS) -c makeparam$(SUFFIX)

makeparam$(SUFFIX): makeparam.F main.F
#
# MIND: I do not have a full dependency list for the include
# and MODULES: here are only the minimal basic dependencies
# if one strucuture is changed then touch_dep must be called
# with the corresponding name of the structure
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
poscar.o: poscar.inc poscar.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.inc wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F

$(OBJ_HIGH):
$(CPP)
$(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
$(CPP)
$(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)

fft3dlib_f77.o: fft3dlib_f77.F
$(CPP)
$(F77) $(FFLAGS_F77) -c $*$(SUFFIX)

.F.o:
$(CPP)
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
$(CPP)
$(SUFFIX).o:
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)





<span class='smallblacktext'>[ Edited ]</span>

[cpp6f]

Never mind. It appears that someone (who's name I won't mention) put a line in our pbs script that specifically uses another version of mpi's mpiexec if the executable has "vasp5" in it's name. Unfortunately that person is 9 time zones away from me so I'll just have to take my frustration out on my desk