The imagination theory makes a different prediction about how replay will look: when you rest on the couch, your brain should replay the sequence “dog, vase, water “. You know from past experience that dogs are more likely to cause broken vases than broken vases are to cause dogs – and this knowledge can be used to reorganise experience into a more meaningful order.
In deep RL, the large majority of agents have used movie-like replay, because it is easy to implement (the system can simply store events in memory, and play them back as they happened). However, RL researchers have continued to study the possibilities around imagination replay.
Meanwhile in neuroscience, classic theories of replay postulated that movie replay would be useful to strengthen the connections between neurons that represent different events or locations in the order they were experienced. However, there have beenhintsfrom experimental neuroscience that replay might be able to imagine new sequences. The most compellingobservationis that even when rats only experienced two arms of a maze separately, subsequent replay sequences sometimes followed trajectories from one arm into the other.
But studies like this leave open the question of whether replay simply stitches together chunks of experienced sequences, or if it can synthesise new trajectories from whole cloth. Also, rodent experiments have been primarily limited tospatialsequences, but it would be fascinating to know whether humans’ ability to imagine sequences is enriched by our vast reserve of abstractconceptualknowledge.
A new replay experiment in humans
We asked these questions in a set of recentexperimentsperformed jointly between UCL, Oxford, and DeepMind.
In these experiments, we first taught people a rule that defined how a set of objects could interact. The exact rule we used can be found in the paper. But to continue in the language of the “water, vase, dog” example, we can think of the rule as the knowledge that dogs can cause broken vases, and broken vases can cause water on the floor. We then presented these objects to people in ascrambledorder (like “water, vase, dog”). That way, we could ask whether their brains replayed the items in the scrambled order that they experienced, or in the unscrambled order that meaningfully connected the items. They were shown the scrambled sequence and then given five minutes to rest, while sitting in anMEGbrain scanner.
As in previous experiments, fast replay sequences of the objects were evident in the brain recordings. (In yet another example of thevirtuous circlebetween neuroscience and AI, we used machine learning to read out these signatures from cortical activity.) These spontaneous sequences played out rapidly over about a sixth of a second, and contained up to four objects in a row. However, the sequences did not play out in theexperiencedorder (ie, the scrambled order: spilled water ->vase ->dog ). Instead, they played out theunscrambled, meaningful order: dog ->vase ->spilled water. This answers – in the affirmative – the questions of whether replay can imagine new sequences from whole cloth, and whether these sequences are shaped by abstract knowledge.
However, this finding still leaves open the important question ofhowthe brain builds these unscrambled sequences. To try to answer this, we played a second sequence for participants. In this sequence, you walk into your factory and see spilled oil on the floor. You then see a knocked over oil barrel. Finally, you turn to see a guilty robot. To unscramble this sequence, you can use the same kind of knowledge as in the “water, vase, dog” sequence: knowledge that a mobile agent can knock over containers, and those knocked-over containers can spill liquid. Using that knowledge, the second sequence can also be unscrambled: robot ->barrel ->spilled oil.
By showing people multiple sequences with the same structure, we could examine two new types of neural representation. First, the part of the representation that iscommonbetween spilled water and spilled oil. This is anabstractcode for “a spilled liquid”, invariant over whether we’re in the home sequence or the factory sequence. And second, the part of the representation that is common between water, vase and dog. This is an abstract code for “the home sequence,” invariant over which object we’re considering.
We found both of these types of abstract codes in the brain data. And to our surprise, during rest they played out in fast sequences that were precisely coordinated with the spontaneous replay sequences mentioned above. Each object in a replay sequence was preceded slightly by both abstract codes. For example, during a dog, vase, water replay sequence, the representation of “water” was preceded by the codes for “home sequence” and “spilled liquid”.
GIPHY App Key not set. Please check settings