November 11, 2016

Deviations from expectations: detecting moving objects

Pawel Zmarz and Georg Keller at the FMI have identified neurons in the visual cortex that enable the detection of moving objects as we move along. These neurons integrate visual and motor-related input and signal a mismatch when there is a deviation between actual and predicted visual flow. In addition, these neurons cover clearly defined areas of the visual field, conveying highly specific local information.

To appreciate the extent to which our visual system outperforms machines and computer algorithms, we need only look at self-driving cars. These vehicles require a wide array of sensors – including cameras and laser rangefinders – to detect the movement of oncoming cars or pedestrians. In contrast, we can discern cars and pedestrians effortlessly as we move along, easily extracting the relevant information (e.g. an approaching car) from the continuous flow of visual information generated by self-motion (e.g. trees and houses at the side of the road).

However, the mechanisms underlying this extraordinary ability are not clearly understood.

Georg Keller, Junior group leader at the Friedrich Miescher Institute for Biomedical Research (FMI), and one of his PhD students, Pawel Zmarz, have now identified cells in the visual cortex of the mouse that enable the detection of moving objects during self-motion.

In their experimental setup, mice exploring a virtual environment learned to expect a certain flow of visual input induced by self-motion (corresponding to the trees and houses we barely notice as we drive along). Then the visual flow was briefly perturbed (a pedestrian stepping onto the road). As Zmarz and Keller analyzed neuronal activity in the mouse visual cortex, they identified certain neurons that become active exactly at the same time as the expected visual input is disrupted. Zmarz explains: “These neurons integrate visual and motor-related inputs and signal a mismatch as soon as the visual flow diverges from what is predicted based on our movement.” Once again, the visual system is tuned to respond to deviations from expectations.

The scientists also showed that the mismatch neurons have receptive fields – i.e. they cover a clearly defined area of the visual field. Different neurons become active depending on whether a pedestrian walks on the sidewalk on the near right, or a car approaches far off to the left. Keller comments: “These receptive fields have guided our understanding of vision for many years. They allow the detection of features and the location and shape of visual stimuli.” The Keller group’s study has now integrated the concept of receptive fields with the concept of predictive coding: “According to our study, there are neurons in the visual cortex, with clearly defined receptive fields, that integrate what we expect to see with what we actually see. This system is particularly suited to discern moving objects in our visual field during self-motion.”

Original publication
Zmarz P, Keller GB (2016) Mismatch receptive fields in mouse visual cortex. Neuron 2016 Nov [Epub ahead of print]

More about Georg Keller
Georg Keller is a Junior group leader at the FMI. He and his group study the key principles underlying sensory processing in the visual cortex. In 2014, he was awarded an SNSF Starting Grant; these grants were introduced as a transitional measure for the Swiss research community after the exclusion of Switzerland from the European Research Council (ERC) Horizon 2020 program.
» More about Georg Keller

FMI Mismatch neurons in mouse visual cortex respond to local perturbations of visual flow. Heatmap of neuronal responses.