Doesn't seem to be much of an effect. Though there are more predictions made it isn't helping us find a more concrete Tc for these higher Tc materials because of all the uncertainty around the phase transition.
What we should really try is regenerating the training data with more samples so that we can have steadier temperature ramps and more datapoints in and around the critical point.
Kinda! Still needs some MD steps to let the system "thermalize". I don't think it amounts to compute or time savings as you end up doing the same amount of calculations, or more.
This is actually the approach we should have taken from the start, but we should be using the D3 corrections model. There's no additional computation requirements, but yields a more accurate model.
# OLD
from orb_models.forcefield.pretrained import orb_v2
# NEW
from orb_models.forcefield.pretrained import orb_d3_v2We'll need to regenerate the dataset and run evaluations again. This'll take a few days of compute.
In layered superconductors (like cuprates or iron-based superconductors), the interlayer coupling is often dominated by van der Waals forces. Getting these interactions right is crucial.
I suspect these are bad data entries and not bad modeling, due to the synthetic doping process.
After looking at these high-error materials closer, I found that there a common pattern. All but one of them was synth_doped, and there was a specific material, mp-1218930 which was doped 8 different times and accounts for half of the materials in this high error list.
During a lot of testing, I was cutting simulations off like 20 K above Tc just to save compute and so we can start working on the next sample, but it's important to understand the behavior of the model up until room temperature.
Looking at the model predictions for the 200 K above Tc to get to room temperature, we see consistent and stable predictions for the non-superconducting state.
Good result for Bi2Ca1Cu2Sr2O8.195-MP-mp-1218930-synth_doped. We like to see this kind of certainly, both before and after Tc. There's still some region of uncertainty (and other materials are much more uncertain), but above our estimated Tc it's clear that the model is no longer predicting superconductivity.
On the other hand, take a look at this example for Ba2Ca0.8Ce0.2Cu2Tl2O8-MP-mp-6885-synth_doped. There's isn't really a phase transition and we overshoot expected Tc by quite a lot. I'm finding that a number of our highest-Tc materials are getting predictions like this. This is understandable, as the training data is extremely limited, but it's also important that we get this right.
Simulating the material heating up to 300 K, we watch as the probability of the superconducting state gradually and steadily decreases, which makes it hard to choose an exact Tc.
Potentially we could enforce a higher degree of certainty needed to classify as superconducting. Looking at all these high-Tc examples, I'm finding that the true-Tc is right around where we predict superconductivity at a 60% probability. The extra temperature we allow in the distance from 60% to 50% is often 50 K or more, which is also why we're overshooting Tc quite a lot.
For materials with good phase changes, the difference between 50%, 60%, or 70% won't really matter as they'll happen at practically the same temperature because of the steepness of the change in predictions.
By increasing the level of certainty, requiring a higher probability for superconductivity, it's possible to get better results. However, this kind of tuning usually comes at a cost and other materials will suffer poorer results.
Ideally, we'll be able to train a better model that will help us capture the proper phase changes and get back to more step-change like predictions instead of these gradual slopes.
Neural network seems like the way to go. You have a lot more control over your model and the behavior of it.
We should do a proper model search, which is really doable because these are small models (10s of thousands of parameters). So far I'm finding:
A compression model works well. Hidden layers with a decreasing number of nodes like 256 -> 128 -> 64 -> 2.
High dropout seems to help the model make more certain predictions! We find we get more of those sharp phase shifts and less of the gradual descent of probability across a wide range of temperatures. This is especially apparent for predictions above Tc, where the probability for these higher-Tc materials is near 0.
Training longer than expected. Classic Karpathy wisdom, we should ease up on the early stopping and continue trained even when val loss may not be decreasing. Cause of the complexity of the training data and the relative small size of these models, I think we're okay to push the training longer. Looking at metrics, the eval set stays around it's maximum, but train AUC will continue to rise, while train accuracy stays the same. This may indicate that the model learns increasingly confident and well-calibrated predictions for examples it already classifies correctly, rather than changing its decisions about borderline cases. Useful.
We notice how much sharper the phase change is with this model, as well as the certainty of non-superconductivity above Tc.
It should be noted that the models don't really have much data above ~200 K. In the dataset generation where we run MD temperature ramping simulations to get materials at different temperatures, we currently don't go further than 20 K above Tc. Seeing all these solid 0 class predictions in temperature ranges we've never trained on is a good sign.
While we looked at how well the model could predict Tc for materials with true Tc greater than what was seen in the training set, we also want to know how well the model can predict classes of materials it's never seen before. We have data for classes like BaCu, BiCaCu, Al, As, but what if we completely removed one of these classes from the data and tried to predict that class of materials?
In this experiment, we'll filter out all As (Arsenic) materials from the training data and see how well we predict on that class of superconductor.
Result
Does not look good. We're whiffing on almost all of them, often predicting a Tc of 0 when there is some non-zero Tc. Not that the model only predicts 0 Tc, but when it does predict a non-zero Tc, it is a very low, single digit Tc.
I think we've chosen a pretty hard case where we've completely removed any understanding of how As behaves. I imagine the model would still be able to predict new classes, but with compositions of materials it already has some understanding of. More testing on this hypothesis required.