TL;DR:
- Artificial neural networks (ANNs) powering language models like ChatGPT have limitations in terms of power consumption and lack of transparency.
- Hyperdimensional computing offers a novel approach by representing information as hyperdimensional vectors.
- Hypervectors enable symbolic reasoning, composition, and decomposition of concepts through mathematical operations.
- Hyperdimensional computing has shown promise in tasks such as image classification and abstract visual reasoning.
- It offers advantages over traditional computing, including resilience to errors and transparent decision-making processes.
- Hyperdimensional computing holds potential for efficient and robust computing systems, compatible with low-power hardware and in-memory computing.
- The market implications include advancements in AI capabilities, improved hardware efficiency, and greater transparency in decision-making processes.
Main AI News:
Amidst the remarkable achievements of ChatGPT and other large language models, it seems that the underlying artificial neural networks (ANNs) might be heading in the wrong direction, according to experts. These ANNs are not only power-hungry but also lack transparency. The intricate nature of these systems makes it challenging to comprehend their inner workings and understand their remarkable performance. Consequently, analogical reasoning, a key ability of human cognition, remains elusive within these models, as they struggle to utilize symbols to represent objects, concepts, and their interrelationships effectively.
The limitations of ANNs have been evident for some time. Take, for instance, an ANN that distinguishes circles from squares. One approach is to employ two output neurons, one indicating a circle and the other a square. However, if the system needs to discern the color of the shape, such as blue or red, it necessitates the addition of four output neurons—each representing a combination of color and shape. As the number of features increases, the network requires even more neurons to accommodate the complexity.
This method starkly contrasts with how our brains perceive the world, which encompasses a myriad of variations. “You have to propose that, well, you have a neuron for all combinations,” explains Bruno Olshausen, a neuroscientist at the University of California, Berkeley. “So, you’d have in your brain, [say,] a purple Volkswagen detector.”
In contrast, Olshausen and other researchers contend that the brain represents information through the collective activity of numerous neurons. Instead of encoding the perception of a purple Volkswagen within a single neuron, the brain employs the combined actions of thousands of neurons. These neurons, when fired in distinct patterns, can represent different concepts altogether, such as a pink Cadillac.
This novel perspective forms the foundation of hyperdimensional computing—an innovative approach to computation. At its core, hyperdimensional computing represents each piece of information, be it the concept of a car or its various attributes, as a single entity: a hyperdimensional vector.
A vector is essentially an ordered series of numbers. For instance, a 3D vector consists of three numbers representing the coordinates of a point in 3D space. Conversely, a hyperdimensional vector, or hypervector, can be an array comprising tens of thousands of numbers, symbolizing a point in a high-dimensional space. This mathematical construct, along with the algebra that manipulates these hypervectors, is flexible and potent enough to overcome several limitations of current computing systems, leading to a groundbreaking paradigm in artificial intelligence.
Bruno Olshausen exclaims, “This is the thing that I’ve been most excited about, practically in my entire career.” Hyperdimensional computing holds the promise of ushering in an era of efficient, robust computing and completely transparent machine-made decisions.
Unleashing High-Dimensional Spaces
To comprehend the computational possibilities that hypervectors offer, let’s consider images containing red circles and blue squares. To represent the variables SHAPE and COLOR, vectors are necessary. Additionally, vectors representing the potential values for these variables—CIRCLE, SQUARE, BLUE, and RED—are also required.
Crucially, these vectors must be distinct, and their distinctness can be measured through a property called orthogonality, signifying the right angles between vectors. In 3D space, three mutually orthogonal vectors exist: one along the x-axis, another along the y-axis, and the third along the z-axis. In a 10,000-dimensional space, countless nearly orthogonal vectors emerge due to minor deviations from strict orthogonality.
Creating distinct vectors to represent SHAPE, COLOR, CIRCLE, SQUARE, BLUE, and RED involves assigning random vectors. The abundance of nearly orthogonal vectors in high-dimensional spaces makes it highly likely that the assigned vectors will satisfy this criterion. “The ease of making nearly orthogonal vectors is a major reason for using hyperdimensional representation,” explains Pentti Kanerva, a researcher at the Redwood Center for Theoretical Neuroscience at the University of California, Berkeley, in a pivotal paper from 2009.
Kanerva’s paper builds upon the work conducted in the mid-1990s by Kanerva himself and Tony Plate, a former doctoral student of Geoff Hinton at the University of Toronto. Independently, they developed an algebra for manipulating hypervectors and hinted at its potential in high-dimensional computing.
Based on these hypervectors for shapes and colors, the system devised by Kanerva and Plate demonstrates how to manipulate these vectors through specific mathematical operations. These operations correspond to symbolic manipulation of concepts.
The first operation is multiplication—a means of combining ideas. For example, multiplying the SHAPE vector with the CIRCLE vector binds the two together, creating a representation of the concept “SHAPE is CIRCLE.” This newly formed “bound” vector remains nearly orthogonal to both the SHAPE and CIRCLE vectors. Importantly, the individual components of the bound vector remain recoverable, allowing information extraction from these bound vectors. Given a bound vector representing a Volkswagen, it is possible to extract the vector representing its color: PURPLE.
The second operation, addition, generates a new vector that represents the superposition of concepts. By adding two bound vectors, such as “SHAPE is CIRCLE” and “COLOR is RED,” a vector emerges that signifies a circular shape with a red color. Once again, this superposed vector can be deconstructed into its constituent elements.
The third operation involves permutation, which entails rearranging the individual elements of vectors. For example, given a three-dimensional vector labeled x, y, and z, permutation might interchange the value of x with y, y with z, and z with x. “Permutation allows you to build structure,” notes Kanerva. “It allows you to deal with sequences, things that happen one after another.” Suppose two events are represented by hypervectors A and B. Superposing them into a single vector would erase information about the order of events. However, combining addition with permutation preserves the order, enabling the retrieval of events in their correct sequence by reversing the operations.
Together, these three operations have proven sufficient to establish a formal algebra of hypervectors capable of facilitating symbolic reasoning. Despite initially failing to grasp the potential of hyperdimensional computing, even for Bruno Olshausen, the breakthrough came in 2015 when Eric Weiss, one of Olshausen’s students, demonstrated the ability to represent complex images as single hypervectors. These hypervectors encapsulated information about all objects within the image, including their attributes such as color, position, and size.
Olshausen recalls the moment, exclaiming, “I practically fell out of my chair. All of a sudden, the light bulb went on.” Following this revelation, numerous teams embarked on developing hyperdimensional algorithms to replicate simple tasks previously tackled by deep neural networks over two decades ago, such as image classification.
Consider a labeled dataset comprising images of handwritten digits. An algorithm analyzes the features of each image according to a predetermined scheme, creating a hypervector for each image. Subsequently, the hypervectors of all images depicting zero are added together to form a hypervector representing the concept of zero. This process is repeated for all digits, resulting in ten hypervectors, each representing a specific digit.
When presented with an unlabeled image, the algorithm generates a hypervector for that image, which is then compared against the stored class hypervectors. This comparison determines the digit to which the new image bears the closest resemblance.
However, the potential of hyperdimensional computing extends beyond these initial tasks. Its true power lies in the capacity to compose and decompose hypervectors for reasoning. This was exemplified in March, when Abbas Rahimi and his colleagues at IBM Research in Zurich combined hyperdimensional computing with neural networks to tackle a challenging problem in abstract visual reasoning—Raven’s Progressive Matrices. This problem involves selecting the most fitting image from a set of candidates to fill a blank position in a 3-by-3 grid of geometric objects.
Rahimi explains their approach, stating that they first created a dictionary of hypervectors to represent the objects in each image. Each hypervector in the dictionary encapsulates an object and a combination of its attributes. Next, a neural network was trained to analyze an image and generate a bipolar hypervector that closely aligns with a superposition of hypervectors in the dictionary. The generated hypervector captures information about all objects and their attributes within the image. Rahimi refers to this process as guiding the neural network to a meaningful conceptual space.
Once the network produces hypervectors for the context images and candidate options, another algorithm analyzes these hypervectors to create probability distributions regarding the number of objects, their sizes, and other characteristics present in each image. By transforming these probability distributions into hypervectors, algebraic calculations can be employed to predict the most likely candidate image to fill the blank slot.
In their experiments, the team achieved an accuracy of nearly 88% on one set of problems, whereas solutions based solely on neural networks fell short at less than 61% accuracy. Furthermore, for 3-by-3 grids, their system was approximately 250 times faster than traditional methods relying on symbolic logic rules for reasoning, as the latter approach involves searching through a vast rulebook to determine the correct next step.
A Promising Future
Hyperdimensional computing not only empowers symbolic problem-solving but also addresses certain challenges of traditional computing. Modern computers experience rapid performance degradation in the presence of errors, such as random bit flips that cannot be corrected by built-in error-correcting mechanisms. Moreover, these error-correcting mechanisms can incur performance penalties of up to 25%, as highlighted by Xun Jiao, a computer scientist at Villanova University.
Hyperdimensional computing, on the other hand, exhibits greater tolerance for errors. Even if a hypervector suffers numerous random bit flips, it remains close to the original vector, ensuring that reasoning processes using these vectors remain largely unaffected by errors. Jiao’s team demonstrated that hyperdimensional systems are at least ten times more resilient to hardware faults than traditional ANNs, which are already significantly more fault-tolerant than conventional computing architectures. Leveraging this resilience could lead to the development of efficient hardware solutions. As Jiao explains, “We can leverage all [that] resilience to design some efficient hardware.”
Transparency represents another advantage of hyperdimensional computing. The algebraic operations employed in hyperdimensional systems offer clear insights into the decision-making process, allowing for a deep understanding of why a particular answer was chosen. This level of transparency is absent in traditional neural networks. Consequently, researchers like Olshausen and Rahimi are working on hybrid systems that map real-world objects to hypervectors using neural networks, followed by the utilization of hyperdimensional algebra. In Olshausen’s words, “Things like analogical reasoning just fall in your lap. This is what we should expect of any AI system. We should be able to understand it just like we understand an airplane or a television set.”
The advantages of hyperdimensional computing over traditional approaches suggest that it is well-suited for a new generation of highly robust, low-power hardware. Additionally, it aligns with the concept of in-memory computing systems, where data storage and computation occur on the same hardware—an alternative to the inefficient data transfer between the memory and central processing units in von Neumann computers. Some of these emerging devices can operate at low voltages and exhibit analog characteristics, making them energy-efficient but also susceptible to random noise. Traditional computing systems encounter a barrier with such randomness, but hyperdimensional computing can effortlessly overcome it.
Conclusion:
Hyperdimensional computing represents a groundbreaking shift in AI, revolutionizing the business market. Its ability to enable symbolic reasoning, resilience to errors, and transparent decision-making processes opens up new opportunities for efficient and robust computing systems. This development will lead to enhanced AI capabilities, improved hardware efficiency, and greater transparency in decision-making, shaping the future of AI-driven solutions in the business landscape.
