Is this is GP appointment for a new problem, or a consultant appointment for an ongoing issue?
If it's just "my back hurts, I'm going to the doctor" then I wouldn't expect a partner to attend, and him asking to is a bit weird.
If it's an ongoing issue that's effectively a disability, and this appointment is to talk about treatment options, and/or discuss the impact on every day life, then having a partner there might be helpful so they can concentrate on the practical issues, e.g. if you need surgery what will your recovery period look like, etc.
Have you had problems in the past feeling intimidated at medical appointments, or not retaining information (which can often be very confusing)? I used to attend my husband's appointments with him because he had a horrible habit of hearing "There's a 0.1% risk of this happening long term" and come out believing he'd been told he'd be in a wheelchair this time next year. He also had a tendency to throw a tantrum and strop off if he was kept waiting more than 15 mins past his appointment time, despite not needing to be anywhere...
So I think we'd need more context to say whether YABU or YANBU, but I'm sure you can work it out for yourself. At the end of the day, as plenty have said, it's your health and your choice. You have a right to privacy. But if it impacts on family life (potentially), then you may wish to consider bringing him into the appointment or making a voice recording while you're there (I've never had a doctor refuse this.)