I am 57 and had exactly the same experience as you. My boobs have been relatively small and very well-behaved all my life (eg no pain before periods etc.).
I noticed 18 months ago that the left one was significantly bigger and fuller than the other, and made a GP appointment. Saw an absolute muppet locum GP who told me it was mastitis and prescribed antibiotics (it wasn't).
DH who was with me insisted on him referring me for a mammogram straight away which he reluctantly did.
Went for the mammogram next working day (three days later), was told nothing abnormal visible and when I asked what was causing the swelling the unpleasant, rude radiographer was very dismissive and told me there was no explanation.
Google referred only to inflammatory BC which I felt sure it wasn't as I had none of the other symptoms.
Asked for a second opinion at a teaching hospital in a major city. The nice radiographer there detected what she described as 'lots of tiny cysts' and reassured me that this was normal at my age (57).
Saw a better GP who ordered full blood tests, but was never told the results (was with a practice where you couldn't log in to see them).
Went a further eight months with totally uneven boobs, really worried about what was causing it, and felt generally awful.
Made an appointment with the lazy oik who was my nominated GP who squinted at my boobs and said: "I'm guessing it's the left one". No explanation given, no reference made to my blood results, the menopause, anything. If he had noted my blood results, which had 'ABNORMAL' in red in both sections re. hormones, when they were done six months earlier, I would not have had a summer of hell.
Was hit like a train by a bunch of meno symptoms last summer, paid privately for more blood tests and it was revealed that my hormone levels were significantly higher than normal on all counts (which my GP should have mentioned, told me the swelling could have been associated with that, and offered me HRT, but he didn't).
Rang the GP surgery and spoke to a b*tch of a nurse who flatly refused my request for HRT.
I changed GP surgery, was put on combined continuous Kliofem which, within a month, made me feel human again and my boob gradually, very slowly, acknowledged this by deflating back to almost normal size.
Lessons learned = Don't let GP fob you off, insist on a mammogram, ask for bloods for meno if you haven't had them, take HRT if you want to, don't frighten yourself by Googling as I found nothing but reference to IBC.
It's frightening and distressing, and the experience revealed to me the widely-acknowledged fact that many GPs do not know or care about women's issues.
Message me if you want to discuss.
Good luck 
.