The Associative Net

The simple neural network model of heterassociative memory known as the associative net provides the basis for investigating the effects of various biological constraints on associative memory performance.

• The Associative Net
• Towards a Biologically Reasonable Associative Net
• A Stochastic Associative Net
• References
• Background References

The Associative Net

The associative net (Willshaw et al, 1969) consists of a layer of input units connected to a layer of output units by feedforward connections. All unit activities are binary (0 or 1) and all connection weights are also binary. Pairs of patterns are stored in the net using a clipped Hebbian learning rule that changes a connection weight from 0 to 1 if both the input unit and output unit are active for the same pattern pair. The simplicity of this model allows analytical and numerical results to be obtained for finite sized nets using standard probability and information theory. Computer simulation of very large nets (in the order of thousands of input and output units) is also possible. We have used all these approaches in the following research.

Towards a Biologically Reasonable Associative Net

Our work concerns the memory performance of this net when various biologically reasonable criteria are met. These criteria include:

sparse activity

only a few units are active in each pattern

sparse connectivity

an output unit only receives connections from a small number of the input units

noisy input cues

input pattern used during pattern retrieval is a noisy version of a previously stored pattern

probabilistic synaptic transmission

a signal from an input unit only reaches an output unit with a finite probability

In collaboration with Marco Budinich, we have shown how recall performance can be greatly improved when the input cues are noisy by multiple cue presentations (Budinich et al, 1995).

A Stochastic Associative Net

Current work includes looking at memory performance when the transmission of a signal from an input unit to an output unit is probabilistic (Graham and Willshaw, 1997b, 1999). This corresponds to neurobiology - less than half of the action potentials arriving at synapses in the mammalian hippocampus may elicit a postsynaptic response. A stochastic net is formed by treating the connection weights as probabilities of transmission. Instead of being 0 and 1, these weights may now be 0.2 and 0.8, for example. These are the transmission probabilities at a synapse before and after modification by Hebbian learning. Our results indicate that only small differences between these probabilities are required to achieve a functioning associative memory. Differences of the order of 0.4 or less may be optimal if there is a cost associated with the magnitude of the difference.

References

Graham, B. and Willshaw, D. (1999) Probabilistic synaptic transmission in the associative net. Neural Computation, 11(1). (manuscript).

Graham, B. and Willshaw, D. (1997a) Capacity and information efficiency of the associative net. Network, 8, 35-54. (manuscript).

Graham, B. and Willshaw, D. (1997b) An associative memory model with probabilistic synaptic transmission. In Bower, J.M. (ed), Computational Neuroscience: Trends in Research, 1997, 315-319. Plenum Press. (manuscript).

Graham, B. and Willshaw, D. (1996) Information efficiency of the associative net at arbitrary coding rates. Proceedings of ICANN96, 35-40. (manuscript).

Graham, B. and Willshaw, D. (1995a) Improving recall from an associative memory. Biol. Cybern., 72, 337-346. (manuscript).

Graham, B. and Willshaw, D. (1995b) Capacity and information efficiency of a brain-like associative net. In Tesauro, G., Touretzky, D., Leen, T. (eds), Neural Information Processing Systems 7, 513-520. MIT Press. (manuscript).

Budinich, M., Graham, B. and Willshaw, D. (1995) Multiple cueing of an associative net. Int. J. of Neural Systems, Supplementary Issue, 171.

Background References

Buckingham, J. and Willshaw, D. (1993) On setting unit thresholds in an incompletely connected associative net. Network, 4, 441-459.

Buckingham, J. and Willshaw, D. (1992) Performance characteristics of the associative net. Network, 3, 407-414.

Willshaw, D., Buneman, O. and Longuet-Higgins, H. (1969) Non-holographic associative memory. Nature, 222, 960-962.