I don't want to panic you but my advise is always go to the doctor and you may need to be quite insistent. I had a very bad experience with mastitis which led to a very nasty abscess (which is quite rare but nevertheless happens). If the ABs don't work and if at any point you suspect an abscess don't hesitate to shout for an ultrasound.
I had loads of antibiotics, Flucloxacilin, Augmentin and Ciprofloxacilin (or something like that) and none of them worked. They kept sending me away until the thing developed a yellow head and rapidly destroyed breast tissue and skin. The final diameter for the surface wound was 10cm and the hole inside was 3.5x3.5cm. I never had a raised temperature at any point.
I had weeks of packing my leeky wound and couldn't go out the house. It failed to heal properly and I had to have another operation (private this time - what a difference). Still having it packed daily after 4 months! It was August when I had the mastitis/abscess. An ultrasound could have saved all of this.
If it continues not to respond to ABs is still red and inflammed and/or worse a hard collection seems to be forming under the surface then I recommend anyone in this position to insist on an ultrasound. You will be amazed how quickly the thing can spread. Don't take no for an answer. Nobody can give a definitive diagnosis without an ultrasound so they are all just speculating (unless they have x-ray vision).
I am very assertive person but I was fobbed off so many times. Don't let the same thing happen to you.