To be honest, I still have this a little bit. The smaller boob (for some reason) rarely got engorged in the early months, but the bigger one frequently did.
Even now, if DS2 (nearly 9 months) is due a feed from the larger boob you can tell by looking (if I have my top off, you can't tell with a top on) as it is swollen up, but the other boob never increases in size when feed is due (still works perfectly well mind, just not physically noticeable). The gap between the two has lessened over the months, but you can still see it (well, I can).
No explanation for it, but it hasn't hindered breastfeeding in anyway, I just figure I have one boob more inclined towards engorgement than the other. I'd keep nursing from alternate boobs, don't favour the smaller boob it might just lead to you having more discomfort from the larger boob (if you keep missing feeds from that one) it's not necessarily going to increase its size (and you don't want too engorged breasts!)
Also, as I said above you can't tell the size difference when I have a top on, so it's not as though joe public can see - try not to worry too much about it, it will settle down and remember it's likely only you (or any other that sees you in your undies) that is aware of it just now.