SBIR-STTR Award

NGA uses deep networks for many tasks including image registration, land cover segmentation, and object detection. Current deep learning approaches develop specialist networks for each task and type of data. Not only is this inefficient, because networks can’t be reused across tasks, this approach ignores correlations between tasks and data sources that can improve performance. In response, we will develop MultiSCALE, a scale-aware, task-conditioned network designed to operate across multiple, mutually supporting tasks. Multi-image tasks are typically performed by matching the ground sample distance (GSD) of the images, usually through information destroying down-sampling. Instead, MultiSCALE conditions its feature extraction on metadata, such as GSD, viewing angle, solar angle etc. generating features that model these differences. Crucially, rather than train a feature space that is invariant (blind to) to these differences, MultiSCALE’s metadata conditioning makes the network aware of differences, such as absolute size. Cascaded task conditioning will improve performance, for example detecting objects while using land cover to guide the search. MultiSCALE will create a general network, trained to perform many tasks across varying resolution data at their native resolutions, increasing network re-usability and improving task performance to support NGA’s imagery analysis.

In Phase I, our team of Kitware and UC-Berkeley developed Scale-MAE by adding ground sample distance (GSD) to positional encodings, and produced a multiscale representation that achieves state-of-the art results across image classification, semantic segmentation, and object detection tasks. In Phase II, we will create a remote sensing pretraining toolkit to enable fast and easy experimentation with multiple self supervised pertaining techniques that create foundational deep neural network models applicable across NGA. The foundational networks will be tested on the tasks as Scale-MAE in Phase I, and we will also benchmark performance for key point matching. Phase II will extend our Scale-MAE work by integrating additional metadata into the network to increase accuracy by providing it with more information; we will extend the approach to handle inputs including NTM, multi-spectral, and SAR data. Finally, the downstream task networks will be transitioned into NGA SAFFIRE for integration and evaluation. This system will enable NGA to quickly train and deploy new detectors to quickly respond to shifting needs and reduce the time from the analyst’s demand for a new task capability to the execution and availability of said capability.