Scientists Doubt that DeepMind’s AI Is as Good for Fractional-Charge Systems as it Seems

Photo: deepmind.com

In their paper published in Science in 2021, a DeepMind team showed how neural networks can be used to describe electron interactions in chemical systems more accurately than existing methods. A team of researchers from Skoltech, the Zelinsky Institute of Organic Chemistry, HSE University, Yandex, and Kyungpook National University show in their comment in Science that DeepMind AI’s ability to generalise the behaviour of such systems does not follow from the published results and requires revisiting, the Skoltech website says.

Knowing where the electrons are within a molecule can go a long way to explaining its structure, its properties, and its reactivity. Chemists use density functional theory (DFT) methods, approximations to the Schroedinger’s equation, to make accurate and computationally efficient models of molecules and materials. But there are well-known circumstances where DFT tools fail. One is predicting how atoms share electrons; in one famous example, DFT methods incorrectly predict that even when a chlorine and a sodium atom are infinitely far apart, the chlorine atom retains a fraction of one of the sodium atom’s electrons.

Errors like that arise because DFT equations are only approximations of the physical reality. Researchers from the DeepMind machine learning project say that their neural network eliminates that part-of-an-electron error and makes more accurate predictions than traditional DFT methods

Petr Zhyliaev

‘At its core, DFT is a method for solving the Schrödinger equation. Its accuracy is determined by its exchange-correlation part, which is, unfortunately, unknown. To date, over 400 distinct approximations for this part were proposed,’ says Petr Zhyliaev, senior research scientist at Skoltech. ‘One way to build a good exchange-correlation part is to transfer information about it from more “advanced” numerical methods than the density functional theory, which are, however, orders of magnitude less computationally efficient. In their work, DeepMind used a neural network as a universal interpolator to learn the exchange-correlation part of the functional. Their attempt was not the first by far, but is one of the most ambitious.’ 

DeepMind constructed a neural network-based density functional designated as DM21, trained on fractional-electron systems, such as a hydrogen atom with half an electron. To prove its superiority, the authors have tested DM21 on a set of stretched dimers (called a BBB set), eg, two hydrogen atoms at a large distance with a total of one electron.

Relationship between the BBB test systems and the fractional-charge atomics from the training set.

Credit by Michael Medvedev

As expected, the DM21 functional shows an excellent performance on the BBB test set, surpassing by far all the tested classic DFT functionals and DM21m, trained identically to DM21 but without the fractional-electron systems in the training set.

‘Although this may look like DM21 has understood the physics behind the fractional-electron systems, a closer look shows that all the dimers in the BBB set become very similar to the systems in the train set. Indeed, by virtue of electroweak interactions locality, atomic interactions are strong only at short distances, outside of which the two atoms behave essentially as if they were not interacting,’ says Alexander Ryabov, research engineer at Skoltech.

Michael Medvedev

‘In some ways, neural networks are just like humans: they prefer to get the right answer for the wrong reason, rather than the other way around. Therefore, it is not so hard to train a neural network as it is to prove that it has learned the physical laws instead of memorising the right answers. Testing a neural network on systems it has seen during training is akin to examining a schoolboy with a task he has seen a teacher solving just five minutes ago,’ explains Michael Medvedev, Associate Professor at the HSE University Faculty of Chemistry and the leader of Group of Theoretical Chemistry at the Zelinsky Institute of Organic Chemistry of the Russian Academy of Sciences.

Thus, the BBB test set is not a proper one: it does not test DM21 understanding of the fractional -electron systems: DM21 can easily get away with memorising. A thorough analysis of the other four evidences of DM21 handling of such systems did not lead to a decisive conclusion, either. Only its good accuracy on the SIE4x4 set may be ‌reliable—although even there, a clear trend of error growth with distance suggests that DM21 is not completely void of trouble with fractional-electron systems.

The usage of fractional-electrons systems in the training set is not the only novelty in the work by DeepMind. Their idea of introducing physical constraints into a neural network via the training set, as well as the approach for imposing physical sense through training on the correct chemical potential, are likely to be widely used in the construction of neural network DFT functionals in the future.

Read the article on the Skoltech website