In detecting a signal from noise, it is the signal to noise ratio that is important. The window function affects the signal to noise ratio because the measured value at the FFT bin includes noise from the whole bandwidth of the window function's kernel. We need to compare the amount by which the window function attenuates the signal, with the amount of noise the window function collects:
Processing Loss measures the degradation in signal to noise ratio due to the window function. It is the ratio of Coherent Power Gain to Equivalent Noise Bandwidth. For a signal made up of one ideal discrete single line frequency component the Coherent Power Gain is 1 and so the Processing Loss is (not surprisingly) just the reciprocal of the Equivalent Noise Bandwidth. Processing Loss only relates to signal frequency components that happen to fall exactly on an FFT bin. But the model of the the Fourier Transform as a series of filters centred on each FFT bin suggests that frequencies that do not happen to fall exactly on an FFT bin will not be measured at 100% of their full value as the window function's kernel response falls off away from its centre frequency. In fact one can imagine that the measured value will fall off as the actual signal frequency component's frequency moves away from the FFT bin frequency: The effect is to make the measured value dip as the actual frequency moves between FFT bins: this effect is sometimes called the 'picket fence effect' or (because it looks a bit like the edge of a scallop shell) 'scalloping'. It is a reasonable assumption that the worst case occurs when the actual signal frequency falls exactly half way between FFT bins. Scalloping Loss is the apparent attenuation of the measured value for a frequency component that falls exactly half way between FFT bins. It is defined as the ratio of the power gain for a signal frequency component located half way between FFT bins, to the Coherent Power Gain for a signal frequency component located exactly on the FFT bin. Scalloping Loss can be calculated by taking the ratio of the value of the window function's kernel one half a a frequency sample off centre, to its value at the centre. Worst Case Processing Loss is the sum of Processing Loss and Scalloping Loss. This is a measure of the worst case reduction of signal to noise ratio which results from the combination of the window function and the worst case frequency location. It is related to the minim amplitude for a signal frequency component to be detected in broadband noise.
