The main two times to avoid are the TT fortnight and to a lesser extent the Manx GP. For the TT especially, there's little risk of accidentally booking for that time as unless you book close to a year in advance, it's all booked up anyway and prices are very high.
I don't think it ever gets very hot, no more than the odd heatwave like in the UK, but as all the island is either near the sea or in the mountains, it's less extreme than the UK anyway.
I'd look at mid June to mid August, between the motorbike events, or in September. It is nice enough all year round if you're not expecting hot weather and it also doesn't get that cold. I also don't think things shut down in winter, so you'd still be able to get around and visit attractions, albeit could be on a reduced time table.
For the OP, I think if you're relying on public transport, I'd stay in Douglas, then you can get everywhere easily as there will be more buses, otherwise you might have to go via Douglas anyway. With your own car, the south is nice to stay in Port Erin or Port St Mary, can't remember which, they're not far apart and both nice to visit. A good day out is to get the train from the south to Douglas, then the horse drawn tram, then the electric railway and finally the train up the mountain. Then go back the way you came - you can get a ticket that covers all these.
But I'd also visit Peel, Castletown, Ramsey, Douglas and Jurby, all round the island really. I saw seals in the sea off the beach down a little lane near Jurby.
Driving round the glens is also lovely - an advantage of having your own car is you can go anywhere to your own schedule, car hire locally is quite expensive, but both local companies are reliable. I'd expect it is cheaper to drive over with your own car compared with 4 foot passengers or flights and local car hire, but obviously worth checking. Also you'd likely need to pay for parking at whichever UK port you travelled from
I've visited the IoM lots of times for work and have often had a few hours spare where I've travelled around and just seen the sights. A lot of locals don't really explore the island much and consider north of Douglas as 'out in the wilds'. I tell people I've been to Ramsey, Point of Ayre and Jurby all in one day and driven to and from the airport in the south of the island and they pull a face and go 'ooh, I've not been all the way up there in years'.