The RMS detector is best for measuring noise or modulated (especially digitally modulated) signals. This is because the detector averages the power from the RBW output that goes into a display point, which gives a good estimate for these types of signals. Be careful, however, if the bandwidth of the signal and the RBW are less than one display point; in this case the signal is averaged with points that have no signal, giving results that are too low.
RF Detectors 560 & 5400 Series Operation & Maintenance Manual
RF Detectors 70/73/75 Series Instruction Sheet