SBIR-STTR Award

The project proposed will develop a suite of metrics for the evaluation of overall performance of IU systems, as well as for the assessment and diagnosis of individual system components. The metrics will address the issues of the generality and standardization of metrics to permit cross-system comparisons. The dimensions to be investigated for metrics development will include: accuracy, compatibility, conformity, computation efficiency, cost/benefit, ease of use, flexibility, robustness, speed, transportability, and utility. Individual metrics will be developed for the following system components: scene registration, image feature extraction, scene object modeling, the knowledge interface, and the control tecture. The proposed metrics will be reviewed for conformance to technical standards, such as precision and validity, and for overall utility. Detailed procedures will be provided for each metric, to ensure appropriate application. In addition to the metrics themselves we propose to recommend a weighting system, that will permit end-users and developers to aggregate multiple metrics into a single measure, reflecting the importance that they attach to the individual dimensions and components. Finally, we will develop a proposed architecture for a database supporting information access to metrics and procedures, as well as to data from other evaluations, and eventually to benchmark results. There are two major benefits anticipated. The first is the development of a standard set of unbiased metrics that will permit cross-system comparisons. This will be particularly useful for procurement decisions. The second is the development of a set of standard metrics for the evaluation of system components that will permit both performance assessments of separate components, as well as diagnostic evaluation of these components, and identification of failure points and areas most likely to benefit from improvement.

The proposed project will refine and extend methodologies and methods developed in our Phase I proposal for the evaluation of Image Understanding systems so that they can be applied for the overall and diagnostic evaluation of performance of machine vision systems developed under the auspices of the ARPA UGVTEE Phase II Program. This will involve development of a scaleable core set of metrics and methods that can be cost-effectively applied to development efforts. The identification of standard input data and tasks sets and a complete evaluation plan are also proposed. In addition, the project will develop and test metrics and methods that can be used to document minimum technical capabilities for proposals for machine vision applications, again with specific focus on ARPA UGVTEE Phase II Program activities. The goal is to develop metrics which do not unnecessarily restrict design approaches. Modifications to metrics and methods required to extend the utility of these evaluation approaches to the ARPA RADIUS program and to commercial applications of machine vision technology will also be identified. Finally, the project will produce an operational version of the database application demonstrated in our Phase I effort. The database application will provide support to evaluation design and implementation, and facilitate retrieval of evaluation data. It will be implemented on a MAC platform. The proposed metrics and methods will be reviewed for conformance to technical standards, such as precision and validity, for overall utility and for resource requirements. Anticipated

Benefits:
There are four major benefits anticipated: 1) A standard set of evaluation metrics for UGV development. 2) A standard set of metrics for evaluation of the technical capability of proposers in machine vision areas. 3) Identification of modifications required to extend UGV standard metrics for unmanned ground vehicles to both RADIUS and commercial machine vision applications. 4) An operational database to support design, implementation, and review of evaluation data. There are two major benefits anticipated. The first is the development of a standard set of unbiased metrics that will permit cross-system comparisons. This will be particularly useful for procurement decisions. The second is the development of a set of standard metrics for the evaluation of system components that will permit both performance assessments of separate components.