Building on LunaWei's points, the other thing I always check first in this situation is whether the strong signal itself is the limit, not the analyzer's front end. A strong carrier has a phase-noise skirt, and close in — within a few hundred kHz to a few MHz, depending on the source — that skirt can sit above a weak tone no matter how good the analyzer is. If the weak signal is inside that skirt, no amount of attenuation or RBW will reveal it; you need a cleaner source, a notch filter on the strong tone, or a measurement that separates the two in time or frequency.
Second, do the pad test in two steps, not one. Add attenuation and watch both tones: if the weak tone drops dB-for-dB with the pad while the strong tone stays clean, the weak tone is probably real and was just near the noise floor. If the weak tone drops faster, or disappears, it was likely an intermodulation product or a mixer spur in the analyzer, as LunaWei noted. Then remove the pad and add a bandpass or notch filter instead — the filter keeps the strong tone out of the first mixer while preserving sensitivity, which a pad cannot do.
Third, check the analyzer's own third-order intercept and its displayed average noise level at your exact RBW, span and attenuator setting, not the datasheet headline numbers. Dynamic range is a system number at one operating point. A practical sequence that works well is: narrow span, narrow RBW, sample detector for a CW weak tone, preamp only after the front end is proven linear, and a second measurement with a different attenuator setting to confirm the weak tone's level is stable.
If you share the offset between the tones, their levels, and whether the strong signal is CW or modulated, it is much easier to say whether phase noise, intermodulation or displayed noise is the real wall in your case.