Imaging systems comprised of electro-optic (EO) and multiple infrared bands are being deployed due to their ability to provide passive situational awareness that is robust to environmental conditions. Fusing multi-sensor multi-spectral image streams to provide an operator with a single unified image and improving downstream algorithms performance using the fused image is a challenging problem. SPECTACLES will provide a plug and play real-time multi-sensor multi-spectral image fusion system. Inputs are pre-processed to allow varying camera properties and asynchronous image streams to the system. The explicit optimization-based fusion module optimizes across all spectral bands to provide the highest quality image for a human operator and robustness to environmental conditions. An offline optimizer adapts the fusion objective function to improve a specific downstream algorithms performance. SPECTACLES will be demonstrated on recorded datasets and benchmarked against accepted fusion metrics. SPECTACLES inputs and outputs will be compliant with relevant MISB, MISP, and NITFS standards to ease integration and transition to imaging systems of interest for the Navy.
Benefit: SPECTACLES technology provides a plug and play real-time multi-sensor multi-spectral image fusion system that integrates with a wide variety of imaging systems providing an optimized image for an operator and improving the performance of downstream image processing algorithms. SPECTACLES is not tightly coupled to a particular imaging system, which opens numerous opportunities to apply our technology in maritime, aerospace, and space defense applications. Commercial applications, such as medical imaging, agricultural imaging, and environmental remote sensing, deploy similar imaging systems and can also benefit from fusing multiple image sources to improve image quality, robustness to adverse conditions, and higher performance for downstream algorithms (e.x. Object Recognition and Segmentation).
Keywords: Optimal Transport, Optimal Transport, Optimal Mass Transport, EO/IR Fusion, Multi-spectral, Image fusion
