YANBU. What on earth is he doing in your phone anyway? Put a password on it!
I don't let my DH anywhere NEAR my devices. He has form for fucking shit up.. He thinks he's a tech wizard and an IT specialist, because he knows how to uninstall and reinstall apps and operating systems... But I have lost count of the amount of times he has fucked up his own computer, lost all his photos, lost all his files, and had the whole thing crash and burn. This never happens to mine. It's because he's always messing about, and installing different operating systems, and downloading crap, and using VPNs....
He is always complaining that his phone is sluggish too, and keeps stalling, and switching off randomly etc etc, but it's because he is always pissing about with it, and experimenting. Problem is, he's not as clever as he thinks he is with IT and tech, and he keeps breaking stuff/messing it up. I've had the same smartphone since 2019 (the first one I ever had.) We got them at the same time. He is on his 4th one. In 7 years.
When we shared a computer - pre 2010 - I had all of my photos and files and music and documents on it, and he had his, and he fucked up the computer several times, by messing about and experimenting... Including one time when I had spent 8 hours (over 2 days) organising and sorting all my stuff, and he messed up, and had to put the computer back to a restore point from 2 days previously. Undoing everything I had done. He also lost my files a couple of times, and I lost everything. I was in tears... All my documents and music and photos, all gone. (Luckily, my favourite photos were on Facebook, but I lost 90% of my photos., and all the music, and all the documents..)
Once I got my own computer (laptop) in 2010, I never allowed him near it. And I have always been fine, never lost anything, never broke my laptop. HE keeps losing stuff off his, and breaking it, and he can never seem to fathom why. It's like some kind of mental block.
As I said, I never EVER let him near anything of mine, and have my devices locked out with passwords.