I wouldn’t be at all surprised if he did say something like this! He always had a yearning to go and live there, I bet he imagined raising his kids in a “new lives in the wild” kind of way.
And something I’ve noticed about white men who marry non-white women : they LOVE taking about it. (Especially if they’re not known for being politically correct, themselves). It’s like it washes away any hint of racism, past or present.