Revision as of 10:14, 6 November 2024

The Ising machine that thinks

The 2024 Nobel Prize in Physics recognized groundbreaking contributions to machine learning with artificial neural networks (ANNs), specifically honouring John Hopfield and Geoffrey Hinton for their foundational discoveries. This accolade sheds light on the immense progress in neural network research and the pivotal role these advancements play in fields ranging from artificial intelligence to optimization.

This is the first physics prize in the Nobel record that stretches far from the usual remit of physics, to honour instead a topic of computer science. This led to a range of reactions from physicists and computer scientists alike, ranging from physics being in crisis to the Nobel committee being under pressure to recognize the impact of deep learning for otherwise completely useless models, passing by machine learning scooping the Physics Nobel and the usual priority disputes.

The history, the main actor, the developments and the future of neural networks all show, however, that they are deeply rooted in physics and provide inspiration on how those two fields are likely to evolve together.

One of the main actors of neural networks—John Hopfield—has been incorrectly described as a computer scientist, while he is, in fact, a full-fledged physicist, who, furthermore, rooted in physics his approach to the new problems he defined at the interface of neuroscience and computer science. Ironically, in the Molecular Biology Department where he was recruited to expand into neurobiology, he reports that «no one in that department thought of me as anything but a physicist». It is also how he presents himself on social media.

Hopfield is still remembered to this day by physicists for his description of the polariton effect, or the problem of propagation of a photon in a polarizable medium.^[1] This was also described by Pekar and Agranovitch but Hopfield christened the particle and made a lasting impression with what is now known as the Hopfield coefficients, or weights for the fraction of light and matter in their quantum superposition.

This was Hopfield's thesis problem, as formulated for him by Overhauser, who subsequently let him work alone on the topic without any contribution whatsoever. About this major input to traditional physics, which remains in his top 10 most cited papers (with about 10% of the citations from his classic 1982 paper), Hopfield fondly remembers ("Al" is Overhauser):

The single paper written from the 1958 thesis is still highly cited (as is the single author) thanks to the existence of lasers, the polariton condensate, and modern photonics. Thank you, Al. I have done my best to repay you through similarly nurturing another generation of independent students.

—Whatever Happened to Solid State Physics? J. J. Hopfield in Annu. Rev. Condens. Matter Phys. 5:1 (2014).

After this breakthrough that already had made him immortal, Hopfield had the genius intuition to look for a problem tailored to his liking and inspiration. It was not that physics was lacking challenges and important problems, but Hopfield felt that he would be more productive in areas where everything had to be done from scratch. This is Nobel prize level advice: «Acknowledging one’s own abilities, style, and weaknesses is ever so useful.»

His guiding idea at the time, from his solid-state background, was that biological matter was interesting matter on its own, that is to say, interesting from a physics point of view and regardless of its interest for biologists. He started to study hemoglobin for that purpose, and was later recruited by the biologist O Schmitt into his Neuroscience Research Program to study biological information processing instead. This was because Schmitt wanted a physicist in the group, and got Hopfield's name from the iconic John Wheeler (Feynman's doctoral advisor), who (for reasons that Hopfield says he has never grasped) had always been one of his staunch supporters. Hopfield got hooked by the new discipline. He had found, at last, his area of predilection:

How mind emerges from brain is to me the deepest question posed by our humanity.

Hopfied was not the only one of his generation to wander that far beyond his field into biology (a famous example is Leon Cooper, the C of BCS, who turned from superconductivity to the theory of learning in neurobiology). He has been, however, the most successful and impacting. His success has been such as to redefine the frontiers of science:

I am gratified that many—perhaps most—physicists now view the physics of complex systems in general, and biological physics in particular, as members of the family. Physics is a point of view about the world.

It is thus an insight from the Nobel committee to award the first Nobel prize on Artificial Intelligence (there will be many more) to Hopfield, who also was, among other things, president of the American Physical Society. His understanding of what physics is precisely what you would expect from someone who could pierce equally through the wonders of light propagation and how the brain works:

My definition of physics is that physics is not what you’re working on, but how you’re working on it

Hopfield's background in condensed-matter physics was clearly pivotal in his understanding of what we now call Hopfield networks, which are arrays of artificial binary neurons, i.e., variables that can take two values, all interconnected by links, that mimic synapses of biological neurons. This is how Hopfield sketches his network, inline and with three neurons, in his seminal paper:

The idea is that information (three bits stored by A, B and C in this case) can be stored and retrieved in a way similar to how our brain retains memory, in opposition to how a computer RAM physically writes the information at a given location. Instead, the information is stored as stable configurations in a complex configuration space and is retrieved by making it an attractor for other lookalike patterns. This thus performs as an associative memory, as one "reminds" (rather than retrieves) the result by providing a close-enough image of it (rather than its address through a pointer). The memory is encoded by "training" the network, which consists in defining the values for the links (or synapses) that, when the network operates, conditions whether a particular neuron is fired (set to 1) or not through a weighted average of all its connected neurons. The underlying principle is known as Hebbian learning and posits that repeated use strengthens the connections, or that "Neurons that fire together, wire together".

How or why it works is a question of physics. The system looks very much like a classic of statistical physics, the Ising model, that describes magnetism.

[2]')" onmouseout="UnTip()" >

There, instead of neurons, one has spins, and instead of synapses, one has particle interactions. Instead of memories, one has phase transitions, replica symmetry breaking and ferromagnetism. Giorgio Parisi earned the Nobel prize in 2021 for his insights into how such apparently simple systems lead to complex phenomena that stretch the limits of known statistical physics. It is one insight of Hopfield to have perceived, at an early stage, that biology brings a conceptual addition to matter, namely, "function", also of relevance in all fields with an applied/computational character:

The term function is peculiarly biological, occurring in biology and in applied sciences/engineering, which are pursued to benefit humans, but not relevant to pure physics, pure chemistry, astronomy, or geology.

A Hopfield network is basically an Ising model that can be trained to perform a useful function, such as encoding useful patterns ino its inernal strucure. Hopfield indeed highlight his "knowledge of spin-glass lore (thanks to a lifetime of interaction with P.W. Anderson)" in shaping his 1982 paper where he explored emergent collective computational properties through recurrent neural networks​. His model introduced energy minimization concepts and dynamic stability analysis, opening up the ANN framework for both associative memory and complex computational tasks. This brought to a new level earlier ideas in that direction, in particular those of Shun-Ichi Amari who proposed a similar model in the early 1970s for self-organizing nets of threshold elements, investigating their ability to learn patterns and form stable equilibrium states, thereby also functioning as associative memory systems. His work was one of the first to theoretically explore how a network could self-organize, recall patterns, and function as a content-addressable memory. Amari’s research provided early insight into the capabilities of such networks, which could recall entire patterns from partial information—traits we now associate with Hopfield networks. Reacting to the Nobel prize, Amari comments that «Physics is a discipline that originally sought to understand the “laws of matter”, but it has now broadened its scope to include the “laws of information”, which could be called the “laws of things”. Indeed, physics has crossed boundaries.» [1]

While Amari was a visionary, John Hopfield's 1982 model elegantly connected ideas from statistical physics with neurobiology, illustrating how a network of binary neurons could act as a content-addressable memory. The analogy with physical systems such as magnetism or spin-glass theory helped clarify the mathematical underpinnings of how memories could be stored and retrieved from such networks.

One of the most important advancements came from Hopfield's collaboration with David Tank in the mid-1980s. Together, they extended the binary Hopfield network to an analog version, using continuous-time dynamics, which could solve complex discrete optimization problems like the Traveling Salesman Problem (TSP)​. This analog Hopfield network allowed for smoother energy landscapes and more flexible computational capabilities, creating a significant advance in neural computation. Their work on solving TSP through this approach demonstrated the practical applicability of neural networks to complex real-world problems, marking a pioneering moment in optimization theory.

However, the Hopfield-Tank model was not without its critics. In 1988, Wilson and Pawley re-examined the stability of the Hopfield-Tank algorithm when applied to the TSP​. Their findings indicated serious challenges in scaling the algorithm to larger problem sizes, revealing that the model often produced invalid or suboptimal solutions when the number of cities increased. They identified inherent limitations in the model’s ability to handle constraints effectively, especially in the context of analog networks. This critique highlighted that while the Hopfield-Tank approach was revolutionary, it was not without limitations, particularly when it came to real-world scalability. Their analysis underscored the need for further refinements or alternative methods to tackle large-scale optimization problems efficiently.

Hopfield's physics approach was, ironically, still too mathematical, being in essence deterministic. A crucial ingredient of real-world physical systems—noise and fluctuations—was brought in by the genuine computer scientist (and great-great-grandson of the logician George Boole): Geoffrey Hinton. Also labelled a cognitive scientist and cognitive psychologist, Hinton was until recently the brain of Google Brain (now Google AI). He developed and, in particular, made efficient and performant a new class of neural networks that he dubbed Boltzmann machines.^[3] He did this with Terrence Sejnowski, who was previously studying physics... with Hopfield!

This brought forward not only the stochastic approach that upgraded the networks to tackle probability distributions as opposed to strict data, but also the insight of layers with constrained connectivities, giving rise to restricted Boltzmann machines. With this and other game-changing ideas, such as the backpropagation algorithm to optimize the training process, time delay neural network, or the identification of the importance and role of hidden layers to shape what is now known as deep-learning,

rumelhart86a

Upon receiving the prize, Hinton announced that as a young student:

I dropped out of physics after my first year at university because I couldn’t do the complicated math. So getting an award in physics was very surprising to me.

Interestingly, Hinton recently left his position at Google in the face of his concerns regarding artificial intelligence, so as to speak freely about the risks paused to humanity by this quickly developing technology. He confessed that «a part of him now regrets his life's work.»

The revival of Hopfield's ideas thus ties closely with the field of complex networks, where the interplay between optimization algorithms and neural computation has been increasingly integrated into physical systems. These developments have demonstrated the continued relevance of Hopfield’s models, especially when paired with modern hardware capable of implementing such networks in real time. The rise of quantum computing and neuromorphic hardware has further cemented Hopfield networks as practical tools for both combinatorial optimization and learning systems.

The 2024 Nobel Prize not only celebrates Hopfield's seminal contributions but also reaffirms the enduring impact of his work which, in the hands of such as Hinton, flourished into a new technology that may disrupt intellectual in the same way that the disrupted .

From Amari’s early models to Hopfield’s breakthrough applications and modern extensions, neural networks have continuously evolved to meet the growing demands of machine learning and optimization. While critiques like Wilson’s underscore the limitations of early models, modern advances show that these networks, especially in combination with cutting-edge technologies, hold the potential for future breakthroughs in computation and beyond.

As we look ahead, the cross-pollination between physics, computation, and biology, as exemplified by Hopfield’s work, will continue to inspire innovation, bridging the gap between theory and real-world application. The journey from Amari’s self-organizing nets to today’s sophisticated neural architectures reminds us that foundational ideas in science often pave the way for transformative technologies.

References

1. ↑ Theory of the Contribution of Excitons to the Complex Dielectric Constant of Crystals. J. J. Hopfield in Phys. Rev. 112:1555 (1958).
2. ↑ Making digital aquatint with the Ising model. Y. Meurice in Am. J. Phys. 90:87 (2022).
3. ↑ A learning algorithm for Boltzmann machines. D. Ackley, G. Hinton and T. Sejnowski in Cogn. Sci. 9:147 (1985).
4. ↑ ImageNet classification with deep convolutional neural networks. A. Krizhevsky, I. Sutskever and G. E. Hinton in Commun. ACM 60:84 (2017).