On the false alarm probability of the Normalized Matched Filter for off-grid target detection

Off-grid targets are known to induce a mismatch that dramatically impacts the detection probability of the popular Normalized Matched Filter. To overcome this problem, the unknown target parameter is usually estimated through a Maximum Likelihood strategy resulting in a GLRT detection scheme. While the test statistic for the null hypothesis is well known in the on-grid case, the off-grid scenario is more involved and, to the best of our knowledge, no such theoretical result is available. This paper fills this gap by proposing such an expression under circular compound Gaussian noise with known covariance matrix thanks to a geometrical approach.