Spectral Quality Metric Comparison to Analyst Performance

Research Team: John Kerekes, David Messinger, Paul Lee, Emmett Ientilucci and Brian Leahy (B.S. Student)

Project Scope: This project explored the relationship between previously developed spectral quality metrics (SQMs) and observed performance of spectral image analysts in a target detection task. SQMs considered included the General Spectral Utility Metric by Simmons, et al, the Detection Probability metric by Shen, and the Spectral Quality Rating Scale metric by Kerekes and Hsu.

Project Status: In order to control and know the image characteristics precisely, DIRSIG scenes were used into which target vehicles of varying style and paint were inserted. Eleven of the vehicles were white SUVs and were the targets of interest while the other fifteen were confusers.

The images were rendered under the various conditions and a spectral matched filter was applied to the spectral image. Volunteer analysts were then provided a higher resolution panchromatic image and the matched filter output images. An example is shown in Figure 12-1.

Figure 3.12-1: Example panchromatic and matched filter output image provided to the analysts

The analysts were given the task of finding white SUVs in the imagery. To report the results in a uniform manner, the analysts were given a summary sheet to complete for each image pair. On the sheet they were asked to list the pixel location (in the spectral image) and their level of confidence (1 = lowest, 5 = highest) for each suspected white SUV. They were asked to list no more than twenty entries, and were not told how many vehicles were present, nor if there were any decoys in the imagery. The eighteen image pairs were analyzed by three volunteer analysts with six pairs assigned to each analyst.

The analyst reports were compared to the known truth locations of the white SUVs in the imagery. If the analyst reported a target within one pixel of the known location it was scored as a correct decision. If there was no target within one pixel of the reported location it was scored as a false alarm only if the analyst assigned it a level of confidence higher than 1. Table 12-1 contains a summary of the results for each of the eighteen pairs of images. Note that the 2m spectral images contained 400x400=160,000 pixels, the 4m images contained 200 × 200 = 40,000 pixels and the 8m images contained 100 × 100 = 10,000 pixels.

The results confirmed the dominant influence of spatial resolution on the quality of spectral imagery. While the trends were generally consistent between the analyst results and the metric predictions, there was only a loose correlation (R values between 0.51 and 0.78) indicating the metrics are not capturing all of the necessary characteristics of the spectral imagery.

While the significance of these results is limited by the small scale of this experiment, the study does illustrate an approach to use image simulation to study the driving parameters of spectral image quality. In the future it would be desirable to repeat this type of experiment but with many different target configurations and a larger sensor parameter trade space.