Stat inefficiency

[Vladimir_Konjkov]

Hi Mike

I have been looking at an instability in VMC optimization when the sampled configurations are distributed very unevenly. In regions with few configurations, the Jastrow/backflow parameters can fit statistical noise rather than the underlying function, so the optimized function changes noticeably between independent samplings.

My current optimization scheme is varmin → emin → emin → emin. I am considering replacing the initial varmin step with madmin, since MAD is less sensitive to outliers and may produce a more stable starting wave function. The idea is that the subsequent emin steps can then optimize the energy without inheriting spurious features introduced by fitting the variance.

In the paper, this is described as a problem of local minima. The underlying issue is that a large statistical fluctuation in the tail leads to a poor approximation, and the new distribution generated from this incorrect approximation then reinforces the effect, making the instability self-perpetuating.

Vladimir

[Vladimir_Konjkov]

All previous wave-function optimization calculations were performed using CASINO v2.13.944 [Mike Towler] (30 October 2021), or were based on those results and subsequently further optimized using newer versions. I have now repeated all the calculations using only CASINO v3.1.385 (October 2026), and I no longer observe the large deviations in the measured statistical inefficiency.

The measured statistical inefficiency is now in the range of 0.94–1.04, which is essentially consistent with unity. At this point, I assume that the previously observed deviations may have been related to the insufficient stability of the optimization algorithm in the five-year-old version of CASINO.

[Pablo_Lopez_Rios]

Hi Vladimir,

For reference, in production runs I currently use the following optimisation workflow:

Code: Select all

opt_method: emin
%block opt_plan
1 method=madmin maxiter=1
2 method=madmin maxiter=1
3 method=madmin
4
5
6
7
%endblock opt_plan

that is, two cycles of MADmin with a single optimiser iteration, to prevent the parameters from moving much between config regenerations, then one full MADmin, and a few cycles of emin from there. I find this approach to be pretty robust.

Regarding local minima, I've run into issues that I assume come from near-redundant parameters which cause emin to converge slowly to the correct minimum (or get stuck in the wrong minimum). I recently discussed this with Cyrus Umrigar and he seems to run into the slow convergence problem with his code too. I've had similar issues in my 2nd quantised VMC code too, and also when minimising the variance with the linear method in CASINO (i.e., emin with emin_opt_variance=T). So chances are that there is work to be done in the linear method to address the occurrence of local minima.

Best,
Pablo