No he won't. Our cat Ollie was like this, so much so there was uproar from the neighbours when they found out we were moving so we left him with old lady who lived next door but one.
It turned out he'd have breakfast with us, then move on next door for kippers , then to next door but one for lunch, and back to us for tea. It wasn't unknown for me to be walking round to the shop a good 5 minutes walk away to see him on the inside of someone's windowledge! Everyone had a cat bowl etc for him and each thought he was their cat.