SBIR-STTR Award

The ubiquity of image sensors for data collection creates a glut of data, which leads to bottlenecks in the processing capabilities of modern systems. In order to process this data, meticulously labeled datasets are required and that must be reviewed by humans in order to guarantee state-of-the-art performance. In this effort we endeavor to create a system that can automatically exploit salient information in a training set and utilize human capital efficiently to produce accurate models that will identify the target objects. State-of-the-art algorithms for image processing, speech recognition, and object detection often assume an abundance of labeled data. Indeed, the big data domain is where deep learning algorithms are known to outperform its competitors. This domain also helps practitioners to ignore overfitting concerns. For instance, deep neural networks (DNN) are often trained in the over-parameterized regime where the network has more parameters than the size of the training data and is prone to overfitting if the data has outliers. These practices can be detrimental and sub-optimal in the few labels regime, where quality data is a finite resource and we need to extract the most out of our dataset; while preventing overfitting. This proposed effort will explore several strategies to reduce labeled data, find exemplars for training and adapting to models through: 1) exploiting contextual knowledge implicit in the data, 2) exploit the structure in the problem, and 3) search for robust models to address overfitting, 4) knowledge transfer from related domains and 5) semi-supervised learning (SSL) strategies. These will be followed by evaluation on relevant datasets in object detection and classification.

In this proposed Phase II effort we will implement a software framework that will reduce the time required to annotate large image/video datasets by a factor of 100x while also reducing the data necessary to train state of the art computer vision models by up to 80%. Our strategy combines several elements and ideas. First, we have developed a sub-tile based a priori theory of how and why CNNs can overcome the curse of dimensionality. This theory makes several testable hypotheses that have been verified experimentally. Second, we have developed a framework to exploit the vast amounts of publicly available labeled data sets to build CNNs that are fantastically good at detecting small targets in overhead videos and imagery (both EO and IR). And third, Mayachitra has built an active/continuous learning framework called VisionForge, that requires orders of magnitude fewer annotations from analysts to teach CNNs to detect (bounding box) and classify small targets in EO and IR full motion video (FMV). We will leverage Mayachitra's active learning framework, VisionForge, in building our prototype framework. We note that Mayachitra is currently in the process of transitioning some of our technology in VisionForge to contractors in the Navy. Currently, image analysts painstakingly draw highly accurate bounding boxes around each target in every frame of full motion video (FMV) and then assign individual target specific labels to each bounding box. A single minute of FMV can take two whole days of an analyst's time. This observation is based on input from Navy contractors who are actually performing such annotations for training machine learning methods. Moreover data scarcity is a significant problem in most DoD applications, and accurate pixel level annotation is important to extract accurate machine learning models from the available data. In Phase I we demonstrated that there is very good support for a mathematical theory of sub-tiles for deep learning and that it is applicable to image classification, object detection and localization. In Phase II we will develop, based on these theoretical ideas, a fast and accurate tile-based active learning annotation system. In particular we will extend VisionForge with the tile based theory to allow rapid pixel level annotation of objects. The final goal is to deliver a rapid annotation and deep learning system for detecting and classifying small targets on overhead imagery and FMV from both EO and IR sensors. The deliverables, in addition to monthly updates and detailed interim reports, include software prototypes at 6 month intervals that culminate in a prototype that meets the efficiency gains listed above.