Measuring small signals accurately: A practical guide
Measurement of signals close to the your measuring instrument's noise floor is always difficult.
For many years the best Op Amps have had input noise floors of right around 1 nV/√Hz. This makes any measurement significantly below this value very difficult to measure. Likewise, in RF systems the best amplifiers have Noise Figures below 1 dB, this is also difficult to measure as there are not many amplifiers available that have much better noise figure than this.
Averaging our measurements only reduces the variance of what we are measuring, but does not reduce the effective noise or increase real measurement resolution.
There are two methods we can use to overcome this natural limitation however as will be shown below.
First a note: These methods are usually applied to instruments that make some kind of a measurement that involves a frequency response (the Cross Correlation method only works with signals that have had a DFT or FFT applied) . While these methods could be applied to a strictly DC measurement, there are probably better methods, such as adding dither and oversampling.Method #1: Noise De-Embedding
Noise De-Embedding is a method to measure the noise floor of the instrument in question then by using math to get a better accuracy of the actual signal even when the signal is below the instruments noise floor.
In spectrum analyzer circles it is well known that if one measures a signal that is 3dB above the noise floor, then that signal level is exactly at the noise floor. The two equal valued signals add in power to give a 3dB increase above the noise. But if you just used the measured value you would be 3 dB off in your measurement accuracy. This is true for CW signals or if the signals are noise (or digitally modulated, which looks much like random noise).
So by knowing our instrument's noise floor (by measuring it first) and knowing that our measured signal is 3 dB above the noise floor, we know that the signal is actually at the noise floor. This method only works when the instrument noise is uncorrelated with the signals noise - which is usually the case.
This correction method is commonly called "Noise Floor Correction"  or "Spectrum Analyzer Noise De-Embedding"  and is used in many single channel receiver and Spectrum Analyzers. The limit of the extension is usually about 7 dB. At that point the measured signal is only 1dB above the instrument's natural noise floor and the total measurement gets so low above the actual noise floor that it is difficult to detect it with any certainty.
The math is not difficult however, and because this is a scalar process it can be used with any measuring instrument.
In power terms: The internal instrument noise (Ni) adds with the measured noise (Nm) to give a displayed total noise (Nt), where Nt = Ni + Nm.
For example if the the total measured noise Nt, is 2dB (which means that the difference between Nm and Ni is 2 db) then the linear power ratio of 2 dB is 1.58 (= 10^(2/10)).
If we set the normalized internal instrument noise to a value of 1.0, then by subtracting this 1.0 value from the linear ratio leaves us with: 1.58 - 1.0 = 0.58.
Note: We are showing only 2 digit precision, but the calculations are being done on a calculator and between steps we are not losing digits. To follow the results exactly do the math step by step, using the multi-digit results from the previous step as the input to the next step.
We can then calculate the correction factor as: 10*Log10(0.58) = -2.33 dB.
So our total correction on this 2 dB signal is: -2.33 - 2.0 = -4.33 dB
What we were actually measuring when we saw a signal that was apparently 2 dB above the noise floor is a signal that was actually -2.33 dB below the noise floor.
A typical correction factor curve that can be generated for this method is shown in Figure 1.
A downside of this method is that if you mis-measure the instrument's noise floor and it is actually higher than you think it is, then the math "Blows up" and you get a very bad result (Figure 1).
Figure 1 - A plot of the Noise De-Embedding correction factor. The Middle trace (Blue) is the Nominal correction factor. In our example: If we measure a signal 2 dB above the noise floor we can read that the total correction factor is -4.33 dB, which means that the signal measured was really: 2 - 4.33 = -2.33 dB below the noise floor. The Red and Green trace show what happens to the algorithm if the noise floor is higher (Red Trace, bottom trace in the figure) or lower (Green Trace, top trace in the figure) than we think it is. These traces show the effect of a 0.5 dB error in our noise floor estimation. As can be seen, even a 0.5 dB error can add a severe measurement uncertainty to the result. In practice we tend to limit the correction factor to 7 dB as about the maximum reasonable correction possible for this method.
So as with all things that involve a calibration, you need need to know how good your calibration is and how well it holds with time and temperature, etc., or measure the noise floor often.
Many of us that work around RF synthesizer circuits use our Spectrum Analyzers to measure Phase Noise as this is quick and easy and doesn't require the big specialized Phase Noise measurement system. The downside of this is that most of the time our spectrum analyzer's noise floor is about the same as even our low cost synthesizers. By using the Noise De-Embedding method we can effectively extend our measurement accuracy better than 6 dB lower than just using the spectrum analyzer measurement result alone.
Another common use for Noise De-Embedding is in RF Measurement receivers where we measure the signal strength of various signals over the air. If we are measuring a signal very close to the sensitivity limit of our receiver we can apply Noise De-Embedding to improve the accuracy of the measurement result by a significant amount all without increasing the cost of our receiver at all.
As can be seen the method can be used in any system that must measure signals that are very close to or below the measurement noise floor for an immediate increase in measurement accuracy.