I should think if you're both older and having a family together isn't on the cards then as long as you're pretty self sufficient you should be okay. Have you thought about what you might want if things were to progress with somebody? Cohabitation, marriage?
I'm married to a doctor, we met when he was in his final year of medical school. He's almost done training to be a GP, which he chose because of the more family friendly hours. When we had our baby he took six months out as well as my 14m of maternity leave and just did locum shifts a few times per month as he recognised he would miss most of DC's babyhood otherwise.
Not sure I could cope with being married to a hospital consultant though, I'd effectively be a single parent. But if I were in my fifties and it was solely about the relationship rather than having kids as well I think it'd be fine.
My own experience isn't very applicable to your situation but suffice to say I spend a lot of time with doctors and have found them to be largely very kind, well mannered, dedicated and family oriented people. Sure there is infidelity and substance abuse in the profession but it's the same for many, many others. See how it goes and if you're content with your own company and life don't write him off.
I wouldn't accept being treated like a second rate option though, he gets time off and days off like everybody else and you deserve to have dates planned into the diary ahead of time instead of being called up last minute to hang out, I wouldn't tolerate that at all. It's one thing to be called in for an emergency when you are doing something together but other than that he will still have time to plan dates with you, make you feel special and woo you. A lot of doctors end up with massively inflated egos and due to being told they're gods constantly, having always been one of the smartest in their peer group, proud family, saving people's lives, dealing with fawning patients, facing life and death every day they can grow to think they're the big 'I am'. Don't drop plans to see him or ever see him last minute, keep busy with your own stuff, and let him pursue you equally. The ones I know with happy relationships are very much with partners who don't take shit, don't let them bring their ego home, expect them to take the bins out and change the nappies and so forth.