DSOs and noise
I recently had a discussion about measuring noise density (i.e., op amp noise) with a digital oscilloscope. While we certainly use oscilloscopes to measure time domain noise, as can be seen on nearly every op amp data sheet, this discussion centered on using the FFT function that nearly every digital scope now has.
A Step Back
First let's take a look at how traditional spectrum analyzers and FFT analyzers measure noise density (for an in-depth look on this see Reference 1).
A spectrum analyzer usually operates exclusively in a 50-Ohm environment (or 75 Ohms as an option). Hence the measurements are usually in dBm or dB with respect to 1 milliwatt with a 50- or 75-Ohm system.
To measure a signal, you set a marker on the peak and read it. To measure noise however you must activate the "mysterious" noise marker function. Now the spectrum analyzer no longer responds well to peak signals because it may be averaging eight or more adjacent data points to reduce the noise variance.
Important correction factors are added to take into account the Equivalent Noise Bandwidth (ENBW) of the resolution bandwidth filter (RBW filter) and a correction may be made if the spectrum analyzer uses a logging detector (most modern analyzers no longer need this correction). The point is that the manufacturer knows what the correction factors are and applies them based on the current settings, and the result is a marker readout with the units of "dBm/Hz." This is the noise density normalized to a 1-Hz bandwidth.
On an FFT-based analyzer (or dynamic signal analyzer) the same thing happens when measuring a signal - move the marker, read the amplitude. And to measure noise a special marker usually called Power Spectral Density (PSD) is activated. FFT boxes operate a little differently than most RF spectrum analyzers. Since the FFT analyzer is operating in a high-impedance environment it does not measure power in dBm per se - it measures voltages into usually a high impedance (1 megohm). So the noise measurement is in RMS volt units. Usually the noise is expressed in nV/rt-Hz (nanovolts per root hertz bandwidth).
The FFT-based analyzer needed to make the same sorts of corrections to the data to make the proper determination of the noise density. Instead of resolution bandwidth filters an FFT analyzer has windowing functions, and these functions have an ENBW also; likewise the sampling rate and FFT size determine the FFT bin width and this needs to be known to get the proper per root Hertz correction factor.
This measure comes about because noise density is expressed in "proportional to power" units. The following equation is the familiar thermal noise of a resistor, and you will notice that the RMS noise voltage of a resistor is proportional to V2 and V2 is proportional to power:
The well known equation for the thermal noise of a resistor. Where kb is Boltzmann's constant, T is degrees in Kelvin and R is the resistance in Ohms. This equation is normalized to a 1 Hz bandwidth so the units are properly in Volts2 RMS per Hz bandwidth.
The above equation has the units of "per Hz bandwidth" and since that is equal to 1 it is just left off the equation in most textbooks, but it is there none the less. So the units of the equation are really: V2 / Hz
The way that most of the electronics industry has decided to express this is by taking the square root of both sides. Since the equation above was normalized to a 1-Hz bandwidth the end result is V/rt-Hz. And since most of our low-noise electronics is in the neighborhood of 1E-9 V/rt-Hz, we can just conveniently scale everything as nanovolts per root Hertz.
Some academic researchers prefer to leave the noise term as V2 and you see that in many papers, but you can just take the square root to get the more familiar V/rt-Hz.
Converting from one unit of measure to the other
You can easily convert from dBm/Hz to V/rt-Hz since both terms are normalized to 1-Hz bandwidth the conversion is just the same as converting dBm to RMS volts (search the Internet for a gazillion explanations of this procedure if needed).
For instance, 1 nV/rt-Hz equals: -167 dB/Hz in a 50-Ohm system.