The main advantage of the VASP code is a reliable set of pseudopotentials covering the entire periodic table. Earlier this year a new set was released with additional potentials optimised for GW calculations. The old potentials remained largely unchanged according to the release document: "In most cases the potentials are literally identical to the previous releases".
After struggling to understand some peculiar results, I realised that the last statement is not always true. The culprit appears to be related to core charges and corrections, which has changed for quite a few elements even for the LDA/PBE sets.
Using VASP 5.2.12 with identical POSCAR, KPOINT and INCAR files, the following results were obtained for oxygen (triplet state, 500 eV planewave cutoff, PBEsol).
(a) Old POTCAR "PAW_PBE O 08Apr2002"
Atom: -0.974485 eV
Molecule: -9.048636 eV
Binding: -7.099666 eV
(b) New POTCAR "PAW_PBE O 08Apr2002"
Atom: -1.571040 eV
Molecule: -10.239136 eV
Binding: -7.097056 eV
Clearly the energy shifts introduced can cancel for a balanced reaction, but if you mistakenly combine calculations using both sets (which appear the same in the header!), you will run into problems.