Event Based RGB D Perception

Overview of the event based RGB D sensing system, structured light patterns, color reconstruction, and depth sensing

← Back to Featured Projects

Real time pixel level color and depth sensing using an event camera and active structured light.

This project explored how controlled illumination could extend the capabilities of a monochrome event camera beyond conventional motion sensing. A TI LightCrafter 4500 projector was synchronized with the camera to introduce precisely timed color and structured light patterns into the scene.

The resulting event stream was processed to recover pixel level color and depth information from both stationary and moving objects. The completed platform combined sensor configuration, hardware triggering, calibration, data acquisition, and three dimensional reconstruction within a unified real time perception system.

Role Prototype Designer and Researcher

Period 2019 to 2024

Organization MIST Lab, Polytechnique Montréal

Focus Event Vision and 3D Perception

Event Camera Structured Light RGB D Sensing 3D Perception Hardware Synchronization ROS Qt C++

My Contributions

My work covered the design and integration of the complete experimental system, from hardware communication and synchronization to software tools, calibration, and perception processing.

Designed and integrated the event camera and projector sensing architecture.
Developed C++ software for camera configuration, bias control, and event acquisition through the camera API.
Integrated USB event streaming and hardware trigger synchronization between the event camera and projector.
Controlled projection sequences for color and structured light acquisition.
Developed ROS compatible tools for event publication, system configuration, and data processing.
Developed a Qt graphical interface for camera setup, calibration, parameter adjustment, and visualization.
Implemented color reconstruction, depth estimation, calibration, and point cloud generation.
Designed and conducted experiments for performance evaluation and validation.

System Architecture

The system combined controlled illumination, asynchronous visual sensing, hardware synchronization, and real time processing within a unified perception pipeline.

1 Structured Light Projection

2 Event Camera Acquisition

3 Hardware Trigger Synchronization

4 C++ and ROS Processing

5 Color and Depth Reconstruction

6 RGB D Point Cloud Output

Hardware Integration

The monochrome event camera and TI LightCrafter 4500 projector were connected through hardware triggers. The camera provided timing information used to coordinate projection sequences, while event data was transferred to the processing computer through USB.

Software Integration

C++ software managed camera parameters, event acquisition, synchronization, and projector operation. ROS provided the communication framework, while Qt tools supported system setup, calibration, visualization, and experimental control.

Perception Pipeline

Rapidly changing projection patterns generated controlled brightness events. These events were processed to recover color information, estimate depth, and reconstruct colorful three dimensional point clouds.

Results

The completed system demonstrated high speed color and depth sensing while retaining the temporal advantages of event based vision.

1400 fps Equivalent color sensing speed

4 kHz Pixel depth detection rate

Static and Moving Scenes Color and depth measurements for a broad range of scene motion

Pixel Level Reconstruction Independent color and depth measurements using controlled projection patterns

Publications, Code, and Research Communication

Event Based RGB Sensing with Structured Light

IEEE/CVF Winter Conference on Applications of Computer Vision, 2023

Ma thèse en 180 secondes, 2024

Polytechnique Montréal research communication profile presenting Event Based RGB D Perception With Structured Light for a general audience.

View MT180 Profile

Event Based RGB D ROS Repository

Public source code and ROS tools developed for the event camera and structured light perception system.

View GitHub Repository

Event Based Vision for Robot Soccer

Related work applying event cameras to fast object tracking, data collection, camera configuration, and ROS based visualization for robotic soccer.

Project Gallery

Selected images from the sensing platform, software tools, calibration process, and reconstructed outputs.

Event camera and TI LightCrafter 4500 experimental setup — Event camera and structured light projector setup.

ROS and Qt calibration interface — ROS and Qt interface for system configuration and calibration.

Real time color reconstruction using projected RGB patterns. The event camera output is shown on the left, the reconstructed color result on the right, and a conventional frame based video at the bottom for visual reference.

Color reconstruction of a rotating pinwheel at different motion speeds. The sequence first shows the raw event camera output, followed by the reconstructed color results as the rotation speed changes.

Color and depth reconstruction of two balls moving vertically through the scene. A volleyball is shown on the left and a smaller blue ball on the right, demonstrating simultaneous RGB and depth recovery for objects with different sizes, colors, and motion.

Example from an event vision dataset that I recorded during a RoboCup competition, shown at five times slower than the original speed. The event camera output appears on the left, while the camera's native color frame output appears on the right. During fast motion, the frame based recording shows visible blur and delay, while the event stream preserves substantially finer temporal detail. The color image was recorded directly by the camera and was not reconstructed by the RGB D system.