VASP scalability on Nahelem with Gigabit interconnection
Posted: Tue Oct 20, 2009 4:36 pm
Dear All,
We recently bought some two-quad core Nehalem X5550 machines to run VASP 4.6. The Nehalem machines are connected via normal Gigabit Ethernet. We used Centos, ifort 11.1.038, and intel MKL 10.2.2.025. The mpi library we used is openmpi-1.3.3. We are happy to see that VASP runs very fast using 8 cores in a single Nehalem box. However, we are very frustrated to find the scalability cross two or more Nehalem boxes is very bad, i.e. VASP doesn’t scale at all cross boxes. The walltime vs. number of cores for a test system (a 2x2 surface) is shown below:
Cores Time (second)
8 85.78
16 116.93
24 245.06
32 261.96
These results surprised me. Actually, VASP scales pretty well on our AMD Barcelona clusters with Gigabit interconnection.
I am wondering if it is a must to use a low-latency interconnection such as infiniband to get a good scalability on Nehalem cross different boxes.
I appreciate it if anybody can share me some experience to run VASP efficiently on Nehalem with Gigabit interconnection.
Thanks.
We recently bought some two-quad core Nehalem X5550 machines to run VASP 4.6. The Nehalem machines are connected via normal Gigabit Ethernet. We used Centos, ifort 11.1.038, and intel MKL 10.2.2.025. The mpi library we used is openmpi-1.3.3. We are happy to see that VASP runs very fast using 8 cores in a single Nehalem box. However, we are very frustrated to find the scalability cross two or more Nehalem boxes is very bad, i.e. VASP doesn’t scale at all cross boxes. The walltime vs. number of cores for a test system (a 2x2 surface) is shown below:
Cores Time (second)
8 85.78
16 116.93
24 245.06
32 261.96
These results surprised me. Actually, VASP scales pretty well on our AMD Barcelona clusters with Gigabit interconnection.
I am wondering if it is a must to use a low-latency interconnection such as infiniband to get a good scalability on Nehalem cross different boxes.
I appreciate it if anybody can share me some experience to run VASP efficiently on Nehalem with Gigabit interconnection.
Thanks.