Despite a large and growing set of available data sources across many domains, identifying and tracking developing threats in operational domains remains a critical unsolved problem. Extraction of relevant threat information and correlation across multiple modalities is required to take full advantage of the data available. VSI proposes TALON, a system for leveraging natural language descriptions of multi-int data, which are semantically rich, extremely flexible, and can be readily generated and processed thanks to recent advances in machine learning, computer vision, and natural language processing. Combined with geo-position and high-value individual (HVI) identifiers, natural language metadata can express a detailed description of virtually any activity or event with its surrounding context. The ability to generate semantically rich natural language-based metadata from multi-modal input source data provides a massive opportunity to leverage higher level reasoning and trend analysis tools originally designed for streams of text. The proposed threat identification system includes a suite of metadata generation algorithms based on state-of-the-art computer vision, machine learning, and natural language processing technology designed to extract detailed content descriptions from multimedia source data. Potential threats are identified within the generated metadata stream by the proposed anomalous activity detection system based on deviations from dynamically learned normalcy models. In addition to developing algorithms for automatic metadata generation and threat analysis, significant effort will be devoted to the curation of rich multi-modal datasets and quantitative evaluation metrics. These datasets and evaluation metrics will be leveraged to establish feasibility of the proposed approach and identify key challenges to be addressed in Phase II. VSI and STR are uniquely suited to successfully execute this project thanks to their experience and expertise in computer vision, remote sensing, natural language processing, and data fusion. VSI and STR have an extensive track record of delivering innovative systems capable of extracting and fusing intelligence from raw data sources such as satellite imagery, news reports, social media, and structured data in challenging real-world problem domains.
Benefit: The proposed Phase I effort establishes feasibility of a threat detection and alerting system capable of ingesting data from several source modalities. Threat detection and alerting is of interest to a variety of government and commercial customers including departments within the Intelligence Community, Department of Defense, law enforcement, and industrial security. The market for tools that automate the labor-intensive process of analyzing large streams of image, text, video, and audio data is large, as the availability of data has far outpaced the ability of human analysts or current state-of-the-art automated threat detection systems to effectively process it. The size of The underlying infrastructure of the proposed system provides an innovative framework to a) generate natural language-based metadata for a variety of data modalities, and b) process the stream of generated metadata to rapidly identify trends, anomalies, and threats. The proposed system is designed to be maximally flexible by nature, allowing it to be adapted for specific use-cases and input data sources as needed. Rigorous test and evaluation on real world data will assert the validity and usefulness of the proposed capability, providing confidence to potential customers.
Keywords: data fusion, data fusion, Natural Language Processing, Machine Learning, threat alerting, activity detection, Multi-int analysis, Computer Vision, Situational Awareness
