Figure: The image on the left shows the sensor setup for the thesis, Sensrad’s 4D imaging radar and an RGB camera mounted on top. On the right is an example visualizing the 4D point cloud and the corresponding camera image pointing to a street crossing with detected objects.

Background

Perception, in the context of object detection for this thesis, is a crucial task that presents challenges depending on the sensor type. Each sensor has its own characteristics, with specific strengths and weaknesses. This makes it beneficial to fuse multiple sensor outputs to detect objects in the surroundings. The radar-camera sensor combination is of particular interest due to the complementary characteristics of these sensors. While a camera excels in recognizing color, texture, and shape, a radar is superior in depth and velocity measurement as well as more robust in challenging lighting and weather conditions. Improving the fusion of these sensors using state-of-the-art algorithms and evaluating the results will be valuable for both industrial applications and academic research.

To conduct the thesis, an internal dataset is available, which can be used in combination with a public dataset like TJ4DRadSet or nuScenes for testing the fusion algorithms. We believe that late fusion architectures, which fuse at the object detection or object track level, could be particularly valuable for industrial applications. Their reduced dependency on specific sensor types makes them especially useful for modular systems. While the fusion algorithms can be based on neural networks, this is not a strict requirement for this thesis. The master’s candidates are expected to work independently, with our guidance, into the topic of cutting-edge fusion algorithms and compare them against each other. In the next step, one or two algorithms should be quantitatively tested and evaluated on the internal and optionally on the public datasets on standard evaluation measures, including prediction accuracy, robustness and real-time performance. In this stage, adjustments to the algorithms and hyperparameter tuning should be made to improve the selected evaluation matrices and fit our radar and camera characteristics. Finally, the candidates should analyze and evaluate the results, especially with focus on the algorithmic capabilities in relation to our sensor setup as well as considering the industrial applicability. The findings of the thesis work should be documented in a report and presented to the company.

This thesis shall consider:

1. Perform a literature study to find the most promising state-of-the-art fusion algorithms that are suitable for our radar-camera setup.
2. Optionally, consider selecting a public data set for the radar camera fusion for object detection in addition to our provided one.
3. Explore ways to improve the given algorithms.
4. Evaluate the quality of the developed fusion method on a test dataset and investigate if the selected method can be run in real-time.

Our radar sensor is one of the most advanced in the world and we think this thesis has the potential to make a significant contribution to both the research community and to our customers. We would therefore be happy if your achievements could lead to a publication and make it into our product.