I don't think this sounds all out terrible if the rest of the relationship is ok (you do go out, you split going out costs, he does bring food, he does help with pets), but it does scream BOUNDARIES and PLANNING (caps for effect 
).
My partner falls squarely into the no-forward-planning, I'll-see-on-the-day category (and he has a child with 50/50 custody, and he takes care of my cat), and at first he'd either turn up or make very loose plans at very short notice, and this would drive me absolutely batty.
I told him that I am an organised person, I like a plan and part of the excitement for me (for a day out, for example) is the anticipation. Just rocking up at midday on a Saturday and saying "what shall we do then?" is a guaranteed way to get my back up.
I understand that there are different personalities at play, but I just made my plans as if I were single. When he called to say "I'll come over tonight", I'd reply "I'm seeing friends/going to the gym/cinema" - even if I wasn't.
There was some initial friction as I actually told him that I did not want to see him unless there was a plan (simply a "would you like to go to the sales this weeknd?", hardly miltary level planning). However, we have found a balance where I'm more chill about specific details, and he's far better at texting/calling and saying "when are you free? do you fancy doing x?".
As for the letting himself in thing and not spending desired alone time actually alone, I'd ask him to stop that. No negotiation, just stop it. Your flat is your space and honestly, it sounds like a bit of a drop-in centre for him when he's sick of his housemates. You're not paying rent/mortgage to provide him with a spare sitting room and a night off. It's cool that he helps with your pets but that is not a free-pass to just use your home as his own.