I wouldn't sacrifice a home I loved for experiences/holidays, no. I have both a home I love and it happens to be big, but I love it because it's given me the lifestyle I want - country living, animals, place for the children to roam, easy access to all the things I love doing, a beautiful garden I enjoy. I wouldn't trade this for a smaller mortgage and more holidays/days out, if that's what your asking.
I live in and enjoy my home and my lifestyle here every day - not just on the weekends, not just once a year for a holiday. But no, I wouldn't want a 'bigger house' just for the sake of it being big, if it didn't give me the location and lifestyle I wanted on a day to day basis.