You will get all sorts of answers on this topic, it's a very personal choice.
In short though, yes, I would. As long as small is comfortable enough to live in (like you said for example a 2 bed flat for a family of 3), I would definitely choose that and do more traveling, have more spare cash to actually live and enjoy my life rather than spend every single penny I have in mortgage repayments and in maintaining a house I can barely afford.