Neuromorphic perception in the loop: simulating event-based sensors forautonomous systems
Type and duration:
Master project, flexible duration.
Context:
Neuromorphic vision sensors, such as event cameras, report changes in
brightness rather than full frames. This promises fast, low-power
perception, especially when some of the processing happens near or on
the sensor itself. But how useful a sensor or preprocessing circuit
really is only becomes clear once it is part of a complete system that
has to act on its output.
In this project you will use simulation to evaluate neuromorphic
perception before hardware is built. You will work with a simulated
robotic platform that has emulated event-based sensors. You will combine
different event-based preprocessing methods with learned controllers,
such as spiking and conventional neural networks trained with imitation
or reinforcement learning, and measure how sensor design choices affect
performance on the task. The aim is to learn what makes a perception
front end useful in a larger system, and to feed those insights into
chip design.
Objectives:
-
Set up and extend a simulation pipeline that pairs emulated event-based sensors with learning-based controllers.
-
Implement and compare different ways of preprocessing event-based sensor data for downstream decision-making.
-
Train neural network controllers, spiking and conventional, on a closed-loop task using imitation and/or reinforcement learning.
-
Systematically evaluate how sensor and preprocessing design choices affect performance and robustness on the task.
-
Turn the findings into concrete recommendations for the design of neuromorphic perception hardware.
Required skills:
You bring: solid Python and experience with PyTorch and machine
learning. An interest in neuromorphic engineering, robotics or
reinforcement learning. C++ and experience with compute clusters are a
plus.
You gain: hands-on experience at the boundary between sensor hardware
and learning systems, on an established simulation platform with cluster
resources.
Contact person:
Hugh Greatorex
References:
-
-