DMC without loadbalancing

[Vladimir_Konjkov]

Hello CASINO developers.
I can suggest a multi-threaded method without loadbalancing для DMC.
This method includes the following steps (for n configurations, m dmc accumulate steps and k processes):
In any process do:
1. create n configurations in VMC run
2. equlibrate n configurations in DMC equlibration run
3. accumulate n configurations over m/k steps
after gather accumulated data from all processes and get mean values (energy etc.)

This method should give the same accuracy as the standard CASINO one.

proc: no loadbalancing needed
con: equlibration required in every process independently

best Vladimir

[Neil Drummond]

I think your suggestion is, basically, to perform k independent DMC simulations with n configurations each and m/k steps.

At one point this was enabled in CASINO (with 1<=k<=P, where P is the number of MPI processes). See the dmc_npops keyword back in CASINO's Git history.

One major con of this approach is that when running on a large number of processes you typically want just a handful of configurations on each process; otherwise you spend all your time equilibrating your massive configuration population. If the populations are independent then you potentially incur significant population-control errors.

Another con: the fractional fluctuations in population decrease with size. So, although you don't have to transfer configurations between processes (or groups of processes), there will be a more uneven load on each process if a population is divided into independent populations due to population fluctuations.

After testing the idea, it was removed from CASINO.

Best wishes,

Neil.

[Neil Drummond]

The other obvious point is that if you just want a series of independent DMC calculations, you can just submit independent jobs with timer-derived random seeds and average the results at the end. If you want to be more careful about non-duplication of random seeds, there is a script called set_random_seed to help with this.

Averaging over repeated DMC calculations is particularly useful if you are averaging over something else at the same time, such as twist angles in periodic calculations, or even nuclear coordinates in vibrationally averaged calculations.

Best wishes,

Neil.

[Vladimir_Konjkov]

Hello Neil.

I want to compare k independent DMC simulations with n configurations each m/n_proc steps vs one DMC simulation with n configurations m steps.
For example n = 1024, m = 100000, n_proc = 32. This requires same time ~n*m/n_proc and gives the same accuracy ~sqrt(n*m), but takes n_proc times more memory and makes each DMC process completely independent after initial configurations generation and excludes loadbalancing.

Simply my python project can't call mpi function from jited code without significant overhead, which interferes with load balancing, so i decided to temporarily abandon this idea and it doesn't seem to make things worse.

Vladimir

[Neil Drummond]

If you do this (scaling down the number of stats accumulation steps for each independent calculation rather than scaling down the target population for each independent calculation) then you will spend nearly all your time equilibrating walkers.

Best wishes,

Neil.