Knitting tension gauge question

[LassoOfTruth]

Sorry this is a boring one but I'm really hoping some more experienced knitters can help! I'm a 'loose' knitter so I always expect that when I can be bothered to do a tension gauge, I will end up going down a needle size to get the correct tension.
Currently attempting a jumper for myself which is composed of panels with 3 different patterns - cable panels back and front, with the shaping/sides in a reverse stocking stitch, sleeves are a modified stocking stitch where every 4th row is reversed (i.e. K instead of P). It calls for knitting tension gauge squares for all 3 of the patterns - so I did. My stocking stitch is actually, pretty well bang on. The cable panel is enormous - I reckon I could maybe resolve it by going down from 5mm to 4.5mm needles, but possibly will even have to use 4mm. The sleeve part, I am my usual looseness - would think I could go down from 5mm to 4.5mm and that should fix it.
So finally then, my question - is this okay? Should I effectively knit this thing using whatever needles I need to get the correct gauge for each pattern part - or should I pick the happy medium to knit the whole thing on the same size needles??

I'd pick the happy medium and use the same needles for all but while also considering that you'll need to sew it all together. Would using the same needles mean that the sleeve might not fit in the armhole?

Love your user name by the way!

Ha thanks!
And thanks do you know I hadn’t even thought about the sewing together. I’ve no time for knitting these days and have stuck to seamless things and baby stuff for so long I’d forgotten the heartbreak of a botched armhole. I might have to modify the pattern itself it it’s coming out all baggy. It’ll be a challenge!

Sewing in sleeves is one of my most hated things in knitting. One thing that's really helped me though is using the little clips that are usually for attaching orchids to support canes. The prong bits are spaced so it doesn't damage the yarn and you don't have to avoid stabbing yourself on pins. Of course it does rely on the sleeve head fitting in the arm hole to begin with!