Compilation Issues with VASP 6.4.2 on AMD EPYC 9654 Using oneAPI.

[hszhao.cn@gmail.com]

Hello,

I am currently attempting to compile VASP 6.4.2 on an AMD EPYC 9654 96-Core processor using the Intel oneAPI 2023.2.0. During the compilation process, I encountered a catastrophic error related to SSE register requirements:

Code: Select all

minimax_functions1D.F(46): catastrophic error: Function return parameter requires SSE register while SSE is disabled.

I understand that this might be related to the specific SSE and AVX instruction sets supported by my processor and below is the flags supported by the processor:

Code: Select all

$ cat /proc/cpuinfo |grep -m1 flags
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d

Based on my tries, to do the complication successfully, the makefile.include.intel needs to be revised as follows, aks, change

Code: Select all

FFLAGS      = -assume byterecl -w

into any of the following ones:

Code: Select all

FFLAGS      = -assume byterecl -w -msse2

or

Code: Select all

FFLAGS      = -assume byterecl -w -mavx512f

So, I ask here for seeking advice on the optimal compiler flags to use for compilation. The goal is to fully leverage the capabilities of the AMD EPYC 9654. Could anyone provide insights or recommendations on how to adjust the compilation flags for VASP 6.4.2 on this architecture? Any other suggestions for resolving the SSE-related error would also be greatly appreciated.

Thank you for your assistance.

Regards,
Zhao

[hszhao.cn@gmail.com]

Here is the AMD EPYC™ 9xx4-series Processors Compiler Options Quick Reference Guide for Intel® oneAPI DPC++/C++ Compiler. Based on the official guidelines above, it seems appropriate to make the following changes:

change

Code: Select all

VASP_TARGET_CPU ?= -xHOST

to

Code: Select all

VASP_TARGET_CPU ?= -axCORE-AVX512

Am I right?

Regards,
Zhao

[hszhao.cn@gmail.com]

After the successful compilation, I ran a test example, the relevant input file is attached. However, I encountered the problem "very serious problems the old and the new charge density differ", as shown below:

Code: Select all

sutai@sutai-Super-Server:~/software/test-large-system$ ./run_vasp.sh 
 
:: initializing oneAPI environment ...
   run_vasp.sh: BASH_VERSION = 5.1.16(1)-release
   args: Using "$@" for setvars.sh arguments: 
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: dal -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dnnl -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: inspector -- latest
:: ipp -- latest
:: ippcp -- latest
:: ipp -- latest
:: itac -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::
 
 running   36 mpi-ranks, on    1 nodes
 distrk:  each k-point on   36 cores,    1 groups
 distr:  one band on    1 cores,   36 groups
 vasp.6.4.2 20Jul23 (build Mar 23 2024 07:46:04) complex                        
  
 POSCAR found type information on POSCAR AuMoS MoS 
 POSCAR found :  5 types and     129 ions
 scaLAPACK will be used
 -----------------------------------------------------------------------------
|                                                                             |
|           W    W    AA    RRRRR   N    N  II  N    N   GGGG   !!!           |
|           W    W   A  A   R    R  NN   N  II  NN   N  G    G  !!!           |
|           W    W  A    A  R    R  N N  N  II  N N  N  G       !!!           |
|           W WW W  AAAAAA  RRRRR   N  N N  II  N  N N  G  GGG   !            |
|           WW  WW  A    A  R   R   N   NN  II  N   NN  G    G                |
|           W    W  A    A  R    R  N    N  II  N    N   GGGG   !!!           |
|                                                                             |
|     For optimal performance we recommend to set                             |
|       NCORE = 2 up to number-of-cores-per-socket                            |
|     NCORE specifies how many cores store one orbital (NPAR=cpu/NCORE).      |
|     This setting can greatly improve the performance of VASP for DFT.       |
|     The default, NCORE=1 might be grossly inefficient on modern             |
|     multi-core architectures or massively parallel machines. Do your        |
|     own testing! More info at https://www.vasp.at/wiki/index.php/NCORE      |
|     Unfortunately you need to use the default for GW and RPA                |
|     calculations (for HF NCORE is supported but not extensively tested      |
|     yet).                                                                   |
|                                                                             |
 -----------------------------------------------------------------------------

 -----------------------------------------------------------------------------
|                                                                             |
|               ----> ADVICE to this user running VASP <----                  |
|                                                                             |
|     You have a (more or less) 'large supercell' and for larger cells it     |
|     might be more efficient to use real-space projection operators.         |
|     Therefore, try LREAL= Auto in the INCAR file.                           |
|     Mind: For very accurate calculation, you might also keep the            |
|     reciprocal projection scheme (i.e. LREAL=.FALSE.).                      |
|                                                                             |
 -----------------------------------------------------------------------------

 LDA part: xc-table for Pade appr. of Perdew
 POSCAR, INCAR and KPOINTS ok, starting setup
 FFT: planning ... GRIDC
 FFT: planning ... GRID_SOFT
 FFT: planning ... GRID
 WAVECAR not read
 entering main loop
       N       E                     dE             d eps       ncg     rms          rms(c)
DAV:   1     0.670920087014E+04    0.67092E+04   -0.36827E+05 14976   0.130E+03
DAV:   2     0.219154142020E+02   -0.66873E+04   -0.63603E+04 19116   0.260E+02
DAV:   3    -0.785483825718E+03   -0.80740E+03   -0.78610E+03 17100   0.909E+01
DAV:   4    -0.829090977810E+03   -0.43607E+02   -0.43104E+02 22428   0.210E+01
DAV:   5    -0.830220500052E+03   -0.11295E+01   -0.11218E+01 21816   0.341E+00    0.142E+02
DAV:   6    -0.786409520503E+03    0.43811E+02   -0.30620E+02 20592   0.188E+01    0.579E+01
DAV:   7    -0.781319533306E+03    0.50900E+01   -0.14160E+02 20808   0.143E+01    0.405E+01
DAV:   8    -0.774189433417E+03    0.71301E+01   -0.21505E+01 19656   0.531E+00    0.926E+00
DAV:   9    -0.774305892561E+03   -0.11646E+00   -0.35321E+00 21636   0.191E+00    0.385E+00
DAV:  10    -0.774164111288E+03    0.14178E+00   -0.68725E-01 20052   0.944E-01    0.179E+00
DAV:  11    -0.774152927940E+03    0.11183E-01   -0.71127E-02 20916   0.437E-01    0.761E-01
DAV:  12    -0.774148986230E+03    0.39417E-02   -0.34878E-02 20520   0.261E-01    0.509E-01
DAV:  13    -0.774147808613E+03    0.11776E-02   -0.57978E-03 20736   0.109E-01    0.259E-01
DAV:  14    -0.774148521832E+03   -0.71322E-03   -0.19314E-03 19224   0.560E-02    0.133E-01
DAV:  15    -0.774150068091E+03   -0.15463E-02   -0.13065E-03 20808   0.480E-02    0.113E-01
DAV:  16    -0.774150890173E+03   -0.82208E-03   -0.25011E-04 19440   0.228E-02    0.758E-02
DAV:  17    -0.774153525448E+03   -0.26353E-02   -0.50376E-04 19980   0.243E-02    0.474E-02
DAV:  18    -0.774155995426E+03   -0.24700E-02   -0.27804E-04 16992   0.197E-02    0.480E-02
DAV:  19    -0.774157548941E+03   -0.15535E-02   -0.10611E-04 17064   0.118E-02    0.240E-02
DAV:  20    -0.774158688166E+03   -0.11392E-02   -0.97539E-05 16812   0.999E-03    0.142E-02
DAV:  21    -0.774159195028E+03   -0.50686E-03   -0.32824E-05 18000   0.652E-03    0.106E-02
DAV:  22    -0.774159448081E+03   -0.25305E-03   -0.56273E-06 16596   0.305E-03    0.746E-03
DAV:  23    -0.774159876059E+03   -0.42798E-03   -0.15716E-05 16668   0.321E-03    0.454E-03
DAV:  24    -0.774160047918E+03   -0.17186E-03   -0.55069E-06 16596   0.197E-03    0.313E-03
DAV:  25    -0.774160134569E+03   -0.86651E-04   -0.18433E-06 13464   0.124E-03    0.196E-03
DAV:  26    -0.774160221425E+03   -0.86856E-04   -0.25693E-06 13896   0.108E-03    0.155E-03
DAV:  27    -0.774160285811E+03   -0.64386E-04   -0.29629E-06 13896   0.911E-04    0.724E-04
DAV:  28    -0.774160293922E+03   -0.81115E-05    0.36308E-07 10944   0.297E-04    0.480E-04
DAV:  29    -0.774160299804E+03   -0.58818E-05    0.40140E-07 10224   0.205E-04    0.363E-04
DAV:  30    -0.774160305719E+03   -0.59151E-05    0.36351E-07 10512   0.186E-04    0.268E-04
DAV:  31    -0.774160307680E+03   -0.19614E-05    0.45904E-07  9576   0.117E-04BRMIX: very serious problems
 the old and the new charge density differ
 old charge density:   954.00000 new  954.07013
    0.843E-02
DAV:  32    -0.781722605202E+03   -0.75623E+01   -0.90896E+00 16848   0.539E-02BRMIX: very serious problems
 the old and the new charge density differ
 old charge density:   954.07013 new  954.45500
    0.408E-01
DAV:  33    -0.826832721363E+03   -0.45110E+02   -0.33552E+01 19116   0.355E-01BRMIX: very serious problems
 the old and the new charge density differ
 old charge density:   954.45500 new  955.04405
    0.974E-01
DAV:  34    -0.881263456074E+03   -0.54431E+02   -0.25548E+01 18756   0.136E+00BRMIX: very serious problems
 the old and the new charge density differ
 old charge density:   955.04405 new  955.84368
    0.154E+00

Why does such a problem occur, whether it is related to the hardware or the system used in the calculation?

Regards,
Zhao

[hszhao.cn@gmail.com]

See the related discussions and solutions below:

https://www.vasp.at/forum/viewtopic.php?t=1417
https://zhuanlan.zhihu.com/p/336754722
https://755452800.github.io/2022/01/29/ ... ty-differ/
https://www.nsc.liu.se/software/install ... lith/vasp/

I have modified the calculation script according to the above suggestions and am currently recalculating. If there are any further results, I will provide timely feedback.

Regards,
Zhao

[andreas.singraber]

Hello Zhao!

(1) Regarding the compilation on 4th generation of AMD EPYC machines: yes, you found the right document from AMD which instructs how to explicitly pass the CPU capabilities to Intel's compiler. I do not have access to such a machine right now but I know that the same issue (catastrophic error: Function return parameter requires SSE register while SSE is disabled.) was present for 2nd and 3rd AMD EPYC generations (I never used 1st gen). The problem is that the Intel compiler does not correctly determine the CPU's instruction (-xHOST) set and automatically falls back to some pre-SSE option for which it is not possible to compile VASP. Hence, we need to explicitly instruct the compiler to generate code for AVX2 (1st to 3rd generation EPYC, -march=core-avx2 or -axCORE-AVX2) or AVX-512 (4th generation EPYC, -axCORE-AVX512).

Now, before applying the new build directly to your system of interest I would strongly recommend to run the testsuite and see if there are any issues!

(2) Concerning the failing example: Did you try this already on a different machine and did it converge there? I see that you get warnings

Code: Select all

WARNING: grid for Broyden might be to small

in your OUTCAR file before the run fails. I am not an expert on convergence issues but maybe it helps if you decrease the mixing parameters (AMIX, BMIX).

All the best,
Andreas Singraber

[andreas.singraber]

Hello again!

I just ran your example on an AMD EPYC 7402 24-core machine with the Intel oneAPI 23.0. Actually, I stopped after the first ionic step.. anyway, I did not find any convergence issues. However, I had to modify your INCAR and KPOINTS settings to speed up the calculation. Instead of

Code: Select all

INCAR
-----
ENCUT = 500
ADDGRID = .TRUE.

KPOINTS
-------
Gamma centered K mesh
0
Gamma
5 5 1 

I used

Code: Select all

INCAR
-----
ENCUT = 300
ADDGRID = .FALSE.

KPOINTS
-------
Gamma centered K mesh
0
Gamma
3 3 1 

Are you sure the computationally expensive settings you provided are absolutely necessary to achieve convergence with respect to your desired property of interest?

Best,
Andreas Singraber

[hszhao.cn@gmail.com]

Dear andreas.singraber,

Thank you very much for your comments and explanations.

Now, before applying the new build directly to your system of interest I would strongly recommend to run the testsuite and see if there are any issues!

The server I purchased is not yet complete and therefore can not be tested systematically. The above tests were conducted using a vendor-provided server for free pretest.

in your OUTCAR file before the run fails. I am not an expert on convergence issues but maybe it helps if you decrease the mixing parameters (AMIX, BMIX).

I don't know either.

Are you sure the computationally expensive settings you provided are absolutely necessary to achieve convergence with respect to your desired property of interest?

I used this example only for rigorous production-level testing to provide objective evaluation criteria for subsequent purchases.

Regards,
Zhao

[hszhao.cn@gmail.com]

Dear andreas.singraber,

Now, before applying the new build directly to your system of interest I would strongly recommend to run the testsuite and see if there are any issues!

Good news, as shown below:

Code: Select all

sutai@sutai-Super-Server:~/software/vasp/vasp.6.4.2/testsuite$ make test
if [ -f tools/compare_numbertable_new ] ; then \
           rm tools/compare_numbertable_new ; fi 
if [ -f tools/m_strings.mod  ] ; then \
           rm tools/m_strings.mod ; fi 
cd tools ;  mpiifort -o compare_numbertable_new compare_numbertable_new.f90
./runtest --all  2>&1 | tee testsuite.log
==================================================================
fatal: 不是 git 仓库（或者任何父目录）：.git
VASP TESTSUITE SHA:
[...]
==================================================================
SUCCESS: ALL SELECTED TESTS PASSED
==================================================================

However, I do not know why the following information appears:

Code: Select all

fatal: 不是 git 仓库（或者任何父目录）：.git

Now, I'm running the more time-expensive testing, aka, `make test_all`. If there is further news, I will give feedback here in time.

Regards,
Zhao

[andreas.singraber]

Dear Zhao,

Great, glad to hear that the testsuite runs without a problem!

You can safely ignore the git-related error message. The released archive is not a git repository (unlike the developer version), hence git commands in the runtest script fail. Maybe we can guard the git commands in future releases to avoid these confusing messages. Thanks for bringing this up!

Best,
Andreas

[hszhao.cn@gmail.com]

Dear andreas.singraber,

Below is the result of make test_all and I also attached the testsuite.log:

Code: Select all

==================================================================
SUMMARY:
==================================================================
The following tests failed, please check the output file manually:
bulk_BN_PBE_Libxc bulk_BN_SCAN_Libxc Co_2E4W_T Co_CRPA_proj Co_CRPA_proj_RPR Co_CRPAR_proj Co_CRPA_weighted Co_CRPA_weighted_RPR FeAl_CRPAR_T mlwf_lif_scdm mlwf_lif_scdm_optics mlwf_mos2_soc_locproj mlwf_mos2_soc_wannier90 mlwf_mos2_wannier90 mlwf_si_scdm NiO_2E4W SrVO3_CRPA_band_removal SrVO3_CRPA_band_removal_RPR SrVO3_CRPA_proj SrVO3_CRPA_proj_RPR SrVO3_CRPA_weighted SrVO3_CRPA_weighted_RPR SrVO3_NLRPA SrVO3_NLRPA_RPR
make: *** [makefile:24：test_all] 错误 1

Upon further analysis of all these errors, it was found that they were due to the lack of the necessary third-party interface libraries required for testing. Therefore, it can be confirmed that this is not an issue with VASP itself:

Code: Select all

$ grep -B2 'I REFUSE'  testsuite.log
|     Error: This functional is not implemented.                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     Error: This functional is not implemented.                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was compiled without the VASP2WANNIER90 interface.                 |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |
--
|     VASP was not compiled with VASP2WANNIER90!                              |
|                                                                             |
|       ---->  I REFUSE TO CONTINUE WITH THIS SICK JOB ... BYE!!! <----       |

[hszhao.cn@gmail.com]

Hello Zhao!

(1) Regarding the compilation on 4th generation of AMD EPYC machines: yes, you found the right document from AMD which instructs how to explicitly pass the CPU capabilities to Intel's compiler. I do not have access to such a machine right now but I know that the same issue (catastrophic error: Function return parameter requires SSE register while SSE is disabled.) was present for 2nd and 3rd AMD EPYC generations (I never used 1st gen). The problem is that the Intel compiler does not correctly determine the CPU's instruction (-xHOST) set and automatically falls back to some pre-SSE option for which it is not possible to compile VASP. Hence, we need to explicitly instruct the compiler to generate code for AVX2 (1st to 3rd generation EPYC, -march=core-avx2 or -axCORE-AVX2) or AVX-512 (4th generation EPYC, -axCORE-AVX512).

[...]

All the best,
Andreas Singraber

A very strange phenomenon regarding the issue above is: although I cannot compile vasp.6.4.2 on AMD CPUs using -xHOST, I found that vasp.6.4.2, which was successfully compiled on Intel CPUs using -xHOST, can be directly used on the AMD EPYC 9004 series. Moreover, it seems to run more efficiently than the corresponding vasp versions locally compiled on the AMD EPYC 9004 series.

Regards,
Zhao

[michael_wolloch]

The intel compiler might not necessarily provide the same type of optimization for all CPUs.

See the line:

Intel optimizations, for Intel compilers or other products, may not optimize to the same degree for non-Intel products.

on https://edc.intel.com/content/www/us/en ... /overview/

[andreas.singraber]

Hello Zhao,

that is very interesting, could you please be more specific? Which machine did you use for compilation and which test did you perform to see the performance difference? Thank you!

All the best,
Andreas Singraber

[hszhao.cn@gmail.com]

Dear Andreas Singraber,

I've sent all my testing examples and results, but your moderator has refused to post such related information.

[michael_wolloch]

Dear Zhao,
sorry for the misunderstanding. I did not approve your post because you linked to a site where a POTCAR file can be downloaded, which is against the VASP license agreement.
I also told you that you should be careful with posting benchmarking results, which is also not allowed without authorization from the VASP company, since benchmarking is a very complex process and hard to do right. But of course, you should be able to report your settings and results of test calculations, so we can help you optimize your compilation setup!

It was not my intention to block you from posting relevant information regarding the test system (just upload your inputs and outputs as per the forum guidelines), as well as compilation settings. Runtime information is included in the OUTCAR.

I apologize for the confusion, I probably was not clear enough in my disapproval message initially.
all the best, Michael Wolloch