The materials for these brief notes come from works [17] and [18]. First of all, it is worth noting that as such there is no single coherent history of Hyperdimensional Computing (HD) or, synonymously,  Vector Symbolic Architecrures (VSA) as different techniques used in HD/VSA have been developed simultaneously and independently. 

HD/VSA is an umbrella term for a family of computational frameworks using high-dimensional distributed representations and relying on the algebraic properties provided by a set of operators (bundling, binding, and permutation) on the high-dimensional representation space. HD/VSA are heavily insrired by D. Gabor’s holographic associative memory [19] and other  models of associative memories inspired by holography [12], [13], [14]. The theory of holography was developed by Gabor in the late 1940s. Psychologists working in the 1960s first suggested holography as a theory of the operation of the brain (see D. Willshaw [20] for a brief review). K. Pribram [21], for instance, suggested that the firing patterns of different neurons may interfere with each other in a manner analogous to the way beams of light interfere, producing hologram-like patterns of interference in the brain. These hologram-like patterns could form the basis of human memory.

Gabor [19] noted that a system that uses the mathematical operations of convolution and cross-correlation can mimic a hologram and could be used as an associative memory. Holography, thus, can be used to define a computational memory system. A. Borsellino and T. Poggio [13] note a formal identity between Gabor’s holographic associative memory [19] and a functional characterization of the neurophysiology of the optomotor response of insects, suggesting that convolution and correlation may indeed be fundamental to neural processes. Cognitive models based on holographic associative memory, such as TODAM by  B. Murdock [10] and CHARM by J. Metcalfe-Eich [11], explain and predict a variety of human memory phenomena.

In a way, HD/VSA are a refinement of Gabor’s holographic associative memory. Research into HD/VSA has been motivated by limitations in the ability of traditional connectionist models (i.e., nonrecurrent models with one or two layers of connections) to represent knowledge with complicated structure [2]. In traditional connectionist models, an item is represented by a pattern of activation across a group of neurons. Mathematically, the pattern of activation is represented by a vector of numbers that stand for the activations of the neurons. Relationships between pairs of items are defined by the connection weights between groups of neurons. The connection weights between two groups of neurons can be represented as a matrix of numbers.

Relationships between more than two items can be defined using more connections, represented as tensors of numbers (i.e., multidimensional arrays; for psychological theory see M. Humphreys, J. Bain, and R. Pike [22]; for computational theory see P. Smolensky [1]). Smolensky’s tensor product binding [1] provides a powerful approach to representing and manipulating relationships between items that differs substantively from traditional connectionist models. The key points demonstrated were that it is possible to represent complex composite data structures (usually thought of as symbolic data structures) in a vector space, and that it is possible to define transformations on that vector space, which manipulate the represented composite structures without needing to decompose them.

Smolensky's method uses the tensor product as the binding operator, which results in the dimensionality of the resultant vector being the product of the dimensionalities of the operand vectors.  In particular, tensor product binding (and, in fact, HD/VSA) do not need to be trained. However, as the number of items bound together into an association grows, the size of the tensor needed to represent the relationship grows exponentially.

HD/VSA are intimately related to tensor product binding [1] and to similar works by E. Mizraji [15], [16]. They provide a means by which relationships between items can be represented using vectors of dimensionality equal to that of the vectors that represent the individual items. In other words, HD/VSA binding operations can be interpreted as forming the tensor product of the operands and then projecting that result back into a vector space with the same dimensionality as each of the operand vectors. Consequently, the dimensionality of the representational space remains constant.

HD/VSA are best understood as a tool for creating and manipulating compact representations of associations that can be situated within a larger framework of structures and processes. As a family, the HD/VSA have been committed to using a fixed dimensionality vector space for the representation and manipulation of composite data structures by exploiting the algebraic properties of a small number of operators on that vector space. Specific HD/VSA frameworks were introduced by T. Plate [2], P. Kanerva [3], R. Gayler [4], D. Rachkovskij [5], and others [6], [7], [8] in the time span of the 1990s to 2000s. The interested readers are kindly referred to [9] where eight different HD/VSA frameworks are compared and taxonomized. 

Starting off with a neurophysiological inspiration that cognitive functions in biological brains are achieved through highly parallel activations of a large number of neurons, which through the learning process form statistically persistent patterns, HD/VSA offer a computational model where everything (items/entities/concepts/symbols/scalars etc.) is represented by  high-dimensional vectors, which act as a distributed representation. In other words, HD/VSA present a plausible means by which the brain could recursively combine patterns of neural activations representing simple concepts to generate new patterns of activation that represent novel, complicated concepts.

HD/VSA provide an extreme form of distributed representation in that it is holographic and stochastic. The holographic property means that the information content of the representation is spread over all the dimensions of the high-dimensional space. Any subset of the dimensions can be used as a noisy approximation to the full set of dimensions. The representations have to be interpreted stochastically, in that the relative proximities of vectors (as measured by the angles between them) are the important properties and the value of any specific dimension makes only a small contribution to the direction of a vector and is consequently nearly irrelevant (in isolation). This is the major departure from the standpoints of the conventional computing, where each component of a representation has a different specific meaning and the exact value of each component is generally important.