Latest revision as of 14:09, 20 November 2024

The Ising machine that thinks

(This piece was written by Natalia Berloff and Fabrice for the HEISINGBERG blog.)

The 2024 Nobel Prize in Physics recognized groundbreaking contributions to machine learning with artificial neural networks (ANNs), specifically honouring John Hopfield and Geoffrey Hinton for their foundational discoveries. This accolade sheds light on the immense progress in neural network research and the pivotal role these advancements play in fields ranging from artificial intelligence to optimization.

This is the first physics prize in the Nobel record that stretches far from the usual remit of physics, to honour instead a topic of computer science. This led to a range of reactions from physicists and computer scientists alike, ranging from physics being in crisis to the Nobel committee being under pressure to recognize the impact of deep learning for otherwise completely useless models, passing by machine learning scooping the Physics Nobel, disappointment by the focus on computational methods and the usual priority disputes.

The history, the main actor, the developments and the future of neural networks all show, however, that they are deeply rooted in physics and provide inspiration on how those two fields are likely to evolve together.

One of the main actors of neural networks—John Hopfield—has been incorrectly described as a computer scientist, while he is, in fact, a full-fledged physicist who furthermore rooted in physics his approach to the new problems he defined at the interface of neuroscience and computer science. Ironically, in the Molecular Biology Department where he was recruited to expand into neurobiology, he reports that «no one in that department thought of me as anything but a physicist». It is also instructive to see how he presents himself on social media:

Hopfield is still remembered to this day by physicists for his description of the polariton effect, or the problem of propagation of a photon in a polarizable medium.^[1] This was also described by Pekar and Agranovitch but Hopfield christened the particle and made a lasting impression with what is now known as the Hopfield coefficients, or weights for the fraction of light and matter in their quantum superposition.

This was Hopfield's thesis problem, as formulated for him by Overhauser, who subsequently let him work alone on the topic without any contribution whatsoever. About this major input to traditional physics, which remains in his top 10 most cited papers (having about 10% citations as compared to his neural network 1982 Nobel-prize paper), Hopfield fondly remembers ("Al" is Overhauser):

The single paper written from the 1958 thesis is still highly cited (as is the single author) thanks to the existence of lasers, the polariton condensate, and modern photonics. Thank you, Al. I have done my best to repay you through similarly nurturing another generation of independent students.^[2]

After this insight that already made him immortal, Hopfield had the genius intuition to look for a problem that would be important yet not so much as to have already attracted enough attention to be—per construct if still unsolved—a very difficult one. He understood that this would bring him beyond his comfort zone and usual community, but still could be tailored to his style and inspiration, and that he would be more productive in areas where everything had to be done from scratch. This is Nobel-prize level advice: «Acknowledging one’s own abilities, style, and weaknesses is ever so useful.»

His guiding idea, from his solid-state background, was that biological matter was interesting matter on its own, that is to say, interesting from a physics point of view and regardless of its interest for biologists. He started to study hemoglobin for that purpose. Although this was probably in itself a good idea that might also have brought him where he is today, he was later diverted by the biologist Francis O Schmitt into biological information processing instead. This was because Schmitt wanted a physicist in his Neuroscience Research Program, and got Hopfield's name from the iconic John Wheeler (Feynman's doctoral advisor), who (for reasons that Hopfield says he has never grasped) had always been one of his «staunch supporters». Hopfield got hooked by the new discipline. He had found, at last, his area of predilection:

How mind emerges from brain is to me the deepest question posed by our humanity.

Hopfied was not the only one of his generation to wander that far beyond his field into biology. A famous example is Leon Cooper, the C of BCS—superconductivity—but also the C of BCM, the theory of synaptic plasticity. He has been, however, the most successful and impacting. His success has been such as to redefine the frontiers of science:

I am gratified that many—perhaps most—physicists now view the physics of complex systems in general, and biological physics in particular, as members of the family. Physics is a point of view about the world.

It is thus an insight from the Nobel committee to award the first Nobel prize on Artificial Intelligence (there will be many more) to Hopfield, who also was, among other things, president of the American Physical Society. His understanding of what physics is, comes as no surprise from someone who could pierce through both the mysteries of light propagation in crystals and how the brain works:

My definition of physics is that physics is not what you’re working on, but how you’re working on it.

Hopfield's background in condensed-matter physics was clearly pivotal in his understanding of what we now call Hopfield networks, which are arrays of artificial binary neurons, i.e., variables that can take two values, all interconnected by links, that mimic synapses of biological neurons. This is how Hopfield sketches his network, inline and with three neurons, in his seminal paper:^[3]

The idea is that information (three bits encoded by A, B and C in this case) can be stored and retrieved in a way similar to how our brain retains memory, in opposition to how a computer RAM physically writes the information at a given location. Instead, the information is stored as stable configurations in a complex configuration space and is retrieved by making it an attractor for other lookalike patterns. This thus performs as an associative memory, as one "reminds" (rather than retrieves) the result by providing a close-enough image of it (rather than its address through a pointer). The memory is encoded by "training" the network, which consists in defining the values for the links (or synapses) that, when the network operates, conditions whether a particular neuron is fired (set to 1) or not through a weighted average of all its connected neurons. The underlying principle is known as Hebbian learning and posits that repeated use strengthens the connections, or that "Neurons that fire together, wire together". Thanks to the nonlinearity of the system, it will bifurcate towards one of the solutions as opposed to returning averages of their correspondences to the input. The model allowed Hopfied to tackle fascinating questions such as the capacity of the memory and its overload, its fidelity, its capacity to discern look-alike memories and the mechanism to do so, its degradation with time and usage, speed of operation, etc.

Winking at the neurobiologist/physics tension in his description, Hopfield gives as an example of something to remember, the reference to a scientific paper. He chose Statistics of the Two-Dimensional Ferromagnet. Part I. H. A. Kramers and G. H. Wannier in Phys. Rev. 60:252 (1941). And argues:

A general content-addressable memory would be capable of retrieving this entire memory item on the basis of sufficient partial information. The input "& Wannier, (1941)" might suffice. An ideal memory could deal with errors and retrieve this reference even from the input "Vannier, (1941)".

Ironically, such skills are required from the reader with his Ref. 26 on previous efforts by Shaw and Roney to find an analytic solution of a neural network theory based on an Ising spin system analogy,^[4] which he incorrectly cites as Phys. Rev. Lett. instead of Phys. Lett. A.

How or why it works is a question of physics indeed. The system looks very much like a famous problem of statistical physics, the Ising model, that describes magnetism.

There, instead of neurons, one has spins (still taking two values, this time, spin up and spin down), and instead of synapses, one has particle interactions. Instead of memories, one has phase transitions, replica symmetry breaking and ferromagnetism. Giorgio Parisi earned the 2021 Nobel prize for his description of how such apparently simple systems can lead to complex phenomena that stretch the limits of known statistical physics. Not surprisingly, Parisi is among those who embrace this year's Nobel prize recognition, observing that «Physics is becoming wider and wider, and it contains many areas of knowledge that did not exist in the past, or were not part of physics.»

It is one insight of Hopfield to have perceived, at an early stage, that biology brings a conceptual addition to matter, namely, "function", also of relevance in all fields with an applied/computational character:

The term function is peculiarly biological, occurring in biology and in applied sciences/engineering, which are pursued to benefit humans, but not relevant to pure physics, pure chemistry, astronomy, or geology.

With this observation, physics grows a pair of wings. A Hopfield network is basically an Ising model that can be trained to perform a useful function, such as encoding useful patterns into its internal structure. Hopfield indeed highlights his «knowledge of spin-glass lore, thanks to a lifetime of interaction with Philip Anderson», the 1977 Physics Nobel prize for the understanding of the structure of magnetic and disordered systems. This shaped his 1982 paper where he explored emergent collective computational properties through recurrent neural networks​. His model introduced energy minimization concepts and dynamic stability analysis, opening up the ANN framework for both associative memory and complex computational tasks.

This brought to a new level earlier ideas in that direction, in particular those of Shun-Ichi Amari who proposed a similar model in the early 1970s for self-organizing nets of threshold elements, investigating their ability to learn patterns and form stable equilibrium states, thereby also functioning as associative memory systems. This work was one of the first to theoretically explore how a network could self-organize, recall patterns, and function as a content-addressable memory. Amari’s research also provided early insight into the capabilities of such networks, which could recall entire patterns from partial information—traits we now associate with Hopfield networks. Reacting to the Nobel prize, Amari comments that «Physics is a discipline that originally sought to understand the “laws of matter”, but it has now broadened its scope to include the “laws of information”, which could be called the “laws of things”. Indeed, physics has crossed boundaries.» [1]

While Amari was a visionary, Hopfield's 1982 model, thanks to its analogy with physical systems such as magnetism or spin-glass theory, did clarify the mathematical underpinnings of how memories could be stored and retrieved from such networks. This elegantly connected ideas from statistical physics with neurobiology, establishing that a simple network of binary neurons could act as a content-addressable memory from sound principles, not as a black box.

One of the most important advancements came from Hopfield's collaboration with David Tank in the mid-1980s.^[5] Together, they extended the binary Hopfield network to an analog version, using continuous-time dynamics, which could solve complex discrete optimization problems like the Traveling Salesman Problem (TSP)​. This analog Hopfield network allowed for smoother energy landscapes and more flexible computational capabilities, creating a significant advance in neural computation. Their work on solving TSP through this approach demonstrated the practical applicability of neural networks to complex real-world problems, marking a pioneering moment in optimization theory.

However, the Hopfield-Tank model was not without its critics. In 1988, Wilson and Pawley re-examined the stability of the Hopfield-Tank algorithm when applied to the TSP​.^[6] Their findings indicated serious challenges in scaling the algorithm to larger problem sizes, revealing that the model often produced invalid or suboptimal solutions when the number of cities increased. They identified inherent limitations in the model's ability to handle constraints effectively, especially in the context of analog networks. This critique highlighted that while the Hopfield-Tank approach was revolutionary, it had serious limitations, particularly when it came to real-world scalability. Their analysis underscored the need for further refinements or alternative methods to tackle large-scale optimization problems efficiently. There were quite a few improvements to these models, in particularly involving the annealing of the gain, and machines based on these principles like the Coherent Ising machine (Stanford and NTT),^[7] Simulated Bifurcation Machine (Toshiba)^[8] or Analog Iterative machine (Microsoft).^[9] These developments opened new directions in analogue computation.

Hopfield's physics approach was, ironically, still too mathematical, being in essence deterministic (although it valued the asynchronicity of neuron-updating as fundamental). A crucial ingredient of real-world physical systems—noise and fluctuations—was brought in by the genuine computer scientist (and great-great-grandson of the logician George Boole): Geoffrey Hinton.

Also labelled a cognitive scientist and cognitive psychologist, Hinton was until recently the brain of Google Brain (now Google AI). He developed and, in particular, made efficient and performant a new class of neural networks that he dubbed Boltzmann machines.^[10] He did this with Terrence Sejnowski, whose background was also in physics. Analyzing the strength of gravitational waves, Sejnowski concluded that the detector efficiency was 1000 times too low and decided to change field. His doctoral advisor was... Hopfield!

Boltzmann machines brought forward not only the stochastic approach that upgraded the networks to tackle probability distributions as opposed to strict data, but also the insight of layers with constrained connectivities, giving rise to so-called restricted Boltzmann machines. The topology of the connectivity was already noted by Hopfield as important in defining the network properties (he took them symmetric to avoid oscillations and chaos). With this and other game-changing ideas, such as the backpropagation algorithm to optimize the training process, time delay neural network, or the identification of the importance and role of hidden layers to shape what is now known as deep-learning,^[11] Hinton transformed Hopfield's ideas from proofs of concepts into a revolutionary technology or, in his own words, «finally something that works well». Today, it powers speech and image recognition with a success that exceeds all expectations. The impact is not only in science anymore but already considerable (and nascent) in industry and tomorrow in society as a whole. His single most cited paper, designing an algorithm able to identify objects in images (AlexNet),^[12] is almost two times more cited than all of Hopfield's papers together.

Hinon's father was a biologist and he says that he was thus more naturally thinking in biological terms. Upon receiving the prize, he recalled his connection to physics:

I dropped out of physics after my first year at university because I couldn’t do the complicated math. So getting an award in physics was very surprising to me.

Interestingly, Hinton recently left his position at Google in the face of his concerns regarding artificial intelligence, so as to speak freely about the risks posed to humanity by this quickly developing technology. He confessed that a part of him now regrets his life's work. Describing the now famous large language model ChatGPT (without which this blog piece was written), he says: «These things are totally different from us. Sometimes I think it’s as if aliens had landed and people haven’t realized because they speak very good English.» With technological progress, Hopfield's sketchy three-neuron idea (he actually simulated 30 neurons with 435 connections and up to 100 neurons in the first paper) can now involve about a trillion connections. That's still a small percentage of the human brain, but GPT-4 already surpasses human capacities in several areas, suggesting that it has a much better learning algorithm. Technology will inevitably make these systems better and ultimately superior to our biological brains. This pauses the new problem for humanity of controlling something more intelligent than itself. This also raises the crucial question of the "compassion" this might have in the likely hypothesis where it might come to form an opinion and/or take action about us. What is especially concerning to Hinton is the immediate and efficient transfer of knowledge and information from one neural net to another. We all know how painful an experience this is for humans. He sees as a critical point of no return the stage where AI will be able to establish its own goals (so far it still mainly "confabulates" or form hallucinations). This is an extract of one of his numerous interviews on the topic, where is upbringing in biology is again obvious:^[13]

Well, here’s a subgoal that almost always helps in biology: get more energy. So the first thing that could happen is these robots are going to say, ‘Let’s get more power. Let’s reroute all the electricity to my chips.’ Another great subgoal would be to make more copies of yourself. Does that sound good?

He concludes the interview with another Nobel-prize level advice: «Enjoy yourself, because you may not have long left.»

The Nobel Prize to artificial intelligence came quicker than most people expected (the 2024 Nobel Prize in Chemistry was also attributed to AI). It also blurs the boundaries between fields, which is probably the sign of a mature science, whose basics (physics, computation, biology, information processing, cognition...) strongly intermingle.

The revival of Hopfield's ideas ties closely with the field of complex networks, where the interplay between optimization algorithms and neural computation has been increasingly integrated into physical systems. These developments have demonstrated the continued relevance of Hopfield’s models, especially when paired with modern hardware capable of implementing such networks in real time. The rise of quantum computing and neuromorphic hardware has further cemented Hopfield networks as practical tools for both combinatorial optimization and learning systems. From Amari’s early models to Hopfield’s breakthrough applications and modern extensions, neural networks have continuously evolved to meet the growing demands of machine learning and optimization. While critiques like Wilson’s underscore the limitations of early models, modern advances show that these networks, especially in combination with cutting-edge technologies, hold the potential for future breakthroughs in computation and beyond.

The 2024 Nobel Prize not only celebrates Hopfield's seminal contributions but also reaffirms the enduring impact of his work which, in the hands of skilled engineers and creative technicians such as Hinton, flourished into a new technology that may disrupt human's intellectual reach in the same way than the steam machine disrupted his mechanical one.

As we look ahead, the cross-pollination between physics, computation, and biology, as exemplified by Hopfield’s work, will continue to inspire innovation, bridging the gap between theory and real-world application. The journey from Amari’s self-organizing nets to today’s sophisticated neural architectures reminds us that foundational ideas in science often pave the way for transformative technologies. The Authors of his blog also nurture ambitions in this direction, namely, to further investigate such wonderful platforms—arrays of interacting variables—with what remains outside of their current dominion: light and quantum mechanics.^[14]

Fabrice P. Laussy and Natalia Berloff.

References

1. ↑ Theory of the Contribution of Excitons to the Complex Dielectric Constant of Crystals. J. J. Hopfield in Phys. Rev. 112:1555 (1958).
2. ↑ Whatever Happened to Solid State Physics? J. J. Hopfield in Annu. Rev. Condens. Matter Phys. 5:1 (2014).
3. ↑ Neural networks and physical systems with emergent collective computational abilities. J. J. Hopfield in Proc. Natl. Acad. Sci. 79:2554 (1982).
4. ↑ Analytic solution of a neural network theory based on an Ising spin system analogy. G. L. Shaw and K. J. Roney in Phys. Lett. A 74:146 (1979).
5. ↑ Computing with Neural Circuits: A Model. J. J. Hopfield and D. W. Tank in Science 233:625 (1986).
6. ↑ On the stability of the Travelling Salesman Problem algorithm of Hopfield and Tank. G. V. Wilson and G. S. Pawley in Biol. Cybern. 58:63 (1988).
7. ↑ Publisher Correction: Coherent Ising machines—optical neural networks operating at the quantum limit. Y. Yamamoto, K. Aihara, T. Leleu, K. Kawarabayashi, S. Kako, M. Fejer, K. Inoue and H. Takesue in npj Quantum Inf. 4:9 (2018).
8. ↑ Combinatorial optimization by simulating adiabatic bifurcations in nonlinear Hamiltonian systems. H. Goto, K. Tatsumura and A. R. Dixon in Science Advances 5:eaav2372 (2019).
9. ↑ Analog Iterative Machine (AIM): using light to solve quadratic optimization problems with mixed variables. K. P. Kalinin, G. Mourgias-Alexandris, H. Ballani, N. G. Berloff, J. H. Clegg, D. Cletheroe, C. Gkantsidis, I. Haller, V. Lyutsarev, F. Parmigiani, L. Pickup and A. Rowstron in arXiv:2304.12594 (2023).
10. ↑ A learning algorithm for Boltzmann machines. D. Ackley, G. Hinton and T. Sejnowski in Cogn. Sci. 9:147 (1985).
11. ↑ Learning representations by back-propagating errors. D. E. Rumelhart, G. E. Hinton and R. J. Williams in Nature 323:533 (1986).
12. ↑ ImageNet classification with deep convolutional neural networks. A. Krizhevsky, I. Sutskever and G. E. Hinton in Commun. ACM 60:84 (2017).
13. ↑ Geoffrey Hinton tells us why he’s now scared of the tech he helped build, interview by Will Douglas Heaven, MIT Technology Review, 2023.
14. ↑ Large-Scale Photonic Ising Machine by Spatial Light Modulation. D. Pierangeli, G. Marcucci and C. Conti in Phys. Rev. Lett. 122:213902 (2019).