Metadynamics issue with OpenMM-PLUMED

[Lili-Summer]

Hello,
I am trying to run MetaD simulation (on Mac M2) with OpenMM-PLUMED. However, despite many efforts I have not been able to run the simulation due to one of the following errors:
1- Looking for a value outside the grid along the 0 dimension
When I check the output file I see at some point CV gets larger than the uwall limit and if I plot "PES vs time" I see sudden spike around the time CV goes out of range.
2- If I increase the GRID_MAX(as one of the suggested solution), then I get another error: open.OpenMMException: Particle coordinate is NaN.
I visually inspected the trajectory (*.dcd) and CB-H bond (ASP137) suddenly behaves poor. I have heavyatom-H constraints. I removed all the constraints and decreased the time step (2 fs to 0.5 fs) but none helped. It seems something fundamental gets broken but I cannot find the source. The tar.gz file is attached to this message
with all necessary files to reproduce the error (run as: nohup ./metaD.sh)
metaD.tar.gz

Versions:
openmm=8.1.1=py311h8ced375_0_khronos
plumed=2.9.0=nompi_haf67379_100
openmmplumed==2.0

I followed this link to install openmm-plumed:
https://github.com/giorginolab/miniomm/wiki/4.-Installing-OpenMM-Plumed-plugin

Thanks in advance for any suggestions/advice!!

[tonigi]

I don't know if it is related to the error, but those instructions may be outdated. There is a conda package now.

[Lili-Summer]

I don't know if it is related to the error, but those instructions may be outdated. There is a conda package now.

I tried conda-forge first but got the error message: "PackagesNotFoundError: The following packages are not available from current channels"
& It seems the conda-forge installation is still not available for Mac M2 (OSX-arm64):
#75

[peastman]

We now have conda packages for ARM Mac. The PR was just merged a couple of days ago.

I'm not really sure what's going on with the original questions. It sounds like possibly more of a PLUMED issue than an OpenMM issue?

[Lili-Summer]

Thanks! I'll install the plugin with conda-forge to see if the problem can be solved.
When I run metaD simulations with OpenMM-PLUMED (with no "upper wall" added) the simulation runs fine (with no error) but then it explores irrelevant parts of phase space (very large distances between atoms of interest). As soon as I add "upper wall", the simulation runs for ~2500-3000 time step (it varies for each run) and then crashes.

I also run the example below (it's supposed to be a working example) & again without "upper wall" the simulation runs fine and with "upper wall" it blows up with NaN value
https://github.com/sukritsingh/openmm-plumed/blob/add-example/examples/basic-example/

& here is the example's plumed-metal.py script with some minor changes ("upper wall added"):
plumed-metad-sim.py.zip

I continued running the example and tried to debug with following some of the suggested solutions: decreased the time step, I removed constraints (heavyatom-H), I removed Barostat and run the simulation as NVT but none helped.

I printed per group energy at the beginning and the time simulations stops:

MINIMIZATION START
initial : -736019.267315558 kJ/mol
final : -990990.2629626248 kJ/mol
MINIMIZATION END

SIMULATION RUN (START)
HarmonicBondForce 662.7587461014557 kJ/mol
HarmonicAngleForce 2844.9604833265676 kJ/mol
PeriodicTorsionForce 12837.33354033684 kJ/mol
NonbondedForce -1007240.7333412252 kJ/mol
CMMotionRemover 0.0 kJ/mol
CustomCVForce 0.0 kJ/mol
PlumedForce 0.0 kJ/mol

SIMULATION RUN (END)
HarmonicBondForce 10388.958613342764 kJ/mol
HarmonicAngleForce 17602.125809446505 kJ/mol
PeriodicTorsionForce 16128.857713073825 kJ/mol
NonbondedForce -834403.6332427938 kJ/mol
CMMotionRemover 0.0 kJ/mol
CustomCVForce 0.0 kJ/mol
PlumedForce 0.0 kJ/mol

I also looked at per atom forces. There are some atoms with large forces (I am lost in the sense that how large it should be to cause a NaN issue).

Is there a way to print per atom forces only for those two atoms in the script?
I run the simulation on CPU (do not have access to GPU at this point). CPU calculates forces with single precision. Can this be the source of issue?
Is it possible to run the example with "upper wall" and "grid_min and grid_max" on your side to see if it runs.
I installed openmm-plumed plugin with SWIG=4.2 and so many OpenMM warnings were generated. SWIG version might have a role?

I really appreciate any suggestion/help so I can pass this step and run the simulations.

[peastman]

Is this the feature you're talking about? I'm not a PLUMED user.

If it works without the upper wall but fails with it, there's a good chance it means the restraining force used to implement the wall is too large. In a typical simulation, most forces are below about 5000 kJ/mol/nm. You have forces four times that large, which could easily cause it to fail.

What about reducing kappa to make the restraining force softer? Whatever value you have right now, try dividing it by 10. Does that bring the forces into a more reasonable range? Does it keep the simulation in the intended area without causing it to fail?

[Lili-Summer]

Yes, that is the feature.
This is the part of script with added wall:

I reduced kappa to 500 & run the simulation. It run for ~18 ps and crashed with NaN. For most part CV is within range (below the upper wall, but there are times that It goes beyond range and wall brings it back).
Visually inspecting the trajectory also shows that around the time CV is out of range noticeably, there is a large force acting on those atoms (residues) and they behave wild (vmd saved sate) is attached (TEST.zip):
TEST.zip
I run another simulation with kappa = 250 and it crashed at ~30 ps with NaN
with kappa = 125, crashed at ~44 ps

Is there any way to select a reasonable value for kappa without doing trial and errors?
& What do I need to consider if I want to switch to WT-metaD regarding the Upper-wall/Lower-wall

again, thanks much for your suggestions/help.

[peastman]

Maybe try reducing EXP to soften it? I'm just guessing from looking at the documentation. I'm not a PLUMED user. I don't know how it's usually chosen.

I notice that example script uses LangevinIntegrator, which is obsolete. If that's what you're using, change it to LangevinMiddleIntegrator. It's more stable.

[Lili-Summer]

Thanks! I'll follow your suggestions.
I am a new user of OpenMM and PLUEMD. They have a User group and I also posted my questions there but have not received any reply yet.
That would be great if you don't close this GitHub issue for a while.

[peastman]

Ok. Let us know what you find.

[Lili-Summer]

Hi @peastman @GiovanniBussi
I do have an update regarding the issue I raised. It seems that the problem is with the latest version of OpenMM-PLUMED plugin (v2).
Another well-experienced user of OpenMM-Plumed raised the same issue (Istvan Kolossvary).
Our simulations run fine with OpenMM7.7, PLUMED 2.7.3 or PLUMED 2.8.1, and Plugin v1 but crash (Particle coordinate NaN) with OpenMM8.1, PLUMED2.9, and Plugin v2.
Giovanni suggested to run the simulations with PLUMED 2.9 and Plugin v1 (Istvan tested this) and majority of simulations finished with no error which suggests that issue is with Plugin v2. Yet, to be confirmed by you.

Thanks much!
Leila

[Alexander-the-Evil]

Hi @peastman @GiovanniBussi @Lili-Summer
I second this issue! I have a weird stuff going on for OpenMM8.1 + Plumed2.9 + Plugin v2.
I run a Chi1-Chi2 torsion metadynamics, and the residue goes haywire. OG-HG bond becomes 1.3 angstroms, though no force should even be applied in that direction.

I would also add that this behavior happens instantly, before Metad even manages to accumulate any sizable bias on CVs. 1 ps is quite enough for it.

My Metad conditions are pretty mild:

METAD ...
  LABEL=M   
  ARG=Chi1,Chi2   
  HEIGHT=0.5   
  PACE=500
  SIGMA=0.35,0.35
  CALC_RCT
  GRID_MIN=-pi,-pi
  GRID_MAX=pi,pi
  GRID_SPACING=0.05,0.05
  BIASFACTOR=8
  TEMP=300
... METAD

Please tell me if any files from my side can be helpful. Otherwise I strongly suggest that some Ala-Ala test runs should be performed.

Thanks!
Alex

[Alexander-the-Evil]

Just checked with OpenMM7.7.0 - Plumed2.7.3 - Plugin v1
Everything is correct chemistry-wise, although the simulation speed is worse but I don't mind as long as it is correct.

Best,
Alex

[gitkol]

Hi,

As Leila mentioned I have run into the (most likely) same problem. Here are the results of my analysis. I ran 6 jobs with the same system and 3 CVs. The CVs are center-of-mass distances between groups of CA atoms. I ran 3 jobs using openmm 7.7, plugin 1.0, and plumed 2.8.1, and another 3 jobs using openmm 8.1.1, plugin 2.0, and plumed 2.9.0. I used 3 different biasing methods. One set of jobs is pure metadynamics (METAD), another set is pure eabf (DRR), and the 3rd set is meta-eabf (DRR on the real CVs and METAD on the fictitious CVs). I attached 3 sets of plots, in each case the top represents plumed v2.9.0 and the bottom v2.8.1. The first is meta, the 2nd is eabf, and the 3rd is meta-eabf. I plotted data at every single time step until the NaNs appeared and eventually the jobs (at the top) crashed. I can also reconfirm that the crashes happened very fast, only after a few hundred time steps.

As you can see, something weird is going on. In the DRR jobs the COM distances computed on the real CV (purple) show some kind of resonance catastrophe until the difference between real and fictitious (green) CVs becomes too big and the force blows up the system. The plain METAD job is different, there are only real CVs, but as you can see, COM-dist1 does a huge jump out of nowhere.

Lastly, I also ran these exact same jobs using openmm 7.7, plugin 1.0, and plumed 2.9.0, but didn't see any weird behavior. So, it seems that the problem might be with plugin 2.0 indeed.

Thanks for looking into this!

Best regards,

Istvan

METAD

EABF

META-EABF

[AlexanderJussupow]

Just a short thanks to the people who found the error source! I had the same issue and could solve it by downgrading openmm-plumed.

[gitkol]

I also wanted to mention that the issue shows up on both M2 and Ubuntu Linux.

[shams-mehdi]

Just wanted to report the same issue i.e, when using plumed for WT-metadynamics, the system starts blowing up immediately after the simulation begins and crashes after ~1 ns. If I downgrade the plugin to v1.0, no crash/bowups.

[peastman]

I've been trying to debug this issue using the example provided by @Lili-Summer. I just can't find anything that OpenMM is doing wrong. PLUMED really is returning huge forces. And if I remove the UPPER_WALLS option, the problem goes away.

Is it possible the problem is caused by using the newer version of PLUMED, not the newer version of OpenMM?

[peastman]

Oh, I found it. After all my searching, it turned out to be something really simple: not clearing a vector before calling PLUMED. The fix is in #82.

[peastman]

New conda packages are now up.

[gitkol]

Thank you, Peter! István

…

On Tue, Apr 2, 2024 at 11:05 AM Peter Eastman ***@***.***> wrote: New conda packages are now up. — Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKDNJ2SAS2LDW3TSKHWCVU3Y3LCLBAVCNFSM6AAAAABCSBP6N2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGMYDGNZTGM> . You are receiving this because you commented.Message ID: ***@***.***>