Many "mystery" measurements come from spectrum analyzer settings rather than the DUT. A practical summary:

Resolution bandwidth (RBW)
- Sets the filter width. Smaller RBW lowers the displayed noise floor (about 10 dB per decade of RBW reduction for noise) and separates close tones, but sweep time increases.
- For noise-like signals (digital modulation, noise), the displayed level depends on RBW. Use channel power or band power measurements, which integrate correctly, rather than reading a marker.

Video bandwidth (VBW)
- Post-detection smoothing. Reduces trace variance but does not lower the true noise floor. VBW much smaller than RBW averages noise; with log scale and averaging this biases noise readings low, which analyzers correct only in specific noise-marker modes.

Detector
- Peak: good for CW tones and finding spurs; overstates noise.
- RMS (average power): correct for power of noise-like signals.
- Sample/normal: various compromises. Standards (EMC, spectrum masks) specify the detector; use what the standard requires.

Input attenuation and reference level
- Too little attenuation drives the first mixer into compression and generates internal distortion that looks like DUT spurs.
- Test: increase attenuation by 10 dB. If a spur drops by 10 dB on screen it is real; if it drops by more, it was generated inside the analyzer.
- Too much attenuation raises the noise floor. Optimize for the measurement (noise floor vs distortion).

Preamplifier
- Lowers the displayed noise floor for weak signals, but reduces headroom.

Other tips
- Use an external reference (10 MHz) when frequency accuracy matters.
- Know the analyzer's phase noise when measuring close-in spurs or ACLR.
- When measuring harmonics, check whether the analyzer's own harmonic distortion is low enough; a filter before the analyzer can help.