As you know, it's a phase and nothing to really worry about, although it can be hurtful. Say something that doesn't minimise what they think they feel while reassuring them of continued love. "Oh, that's a shame, Daddy still loves you and that won't change, even if you don't like him today".
Also something like "will you love Daddy tomorrow?" Can be useful as even if he says no, you can repeat expressions of continued love. As long as they continue to feel safe and secure they'll feel safe telling you how they feel.
Mine are 8 and 5 and there's still the odd flare up of "not you mummy" or "I only want daddy" or "I don't love granny I want her to leave" when my MIL was visiting etc.
Unless it's accompanied by distress at being around the person, it's nothing to fret about too much.
Ds did this. Not a lot you can do other than let it pass. Perhaps a regular 'daddy time' doing something fun just the two of them? I know it doesn't help but it's very normal and healthy and is about your ds's secure attachment to you. Over time he'll form attachments to other trusted adults too. I'm still ds's number 1 but when I told him off yesterday he cried "I want Daddy!!"