It is unlikely that dairy in your diet will be affecting your baby, unless there is a diagnosed allergy there. I was involved in breastfeeding support for years and felt that it is one of those things that people suggest so quickly, and mothers make a herculean effort to eliminate dairy, but usually the only difference is that a) time passes, babies settle a bit whatever you're doing, b) mother suffers and life gets more complicated, and c) bystanders get impression reinforced that BF is difficult and complicated. Yes, some babies do have cow's milk protein allergy, but it's much rarer in breastfed babies than in formula fed (only 0.5% as opposed to around 5% formula fed) . It may be that this is your baby's issue, but I'd try easier steps first - have you seen a breastfeeding counsellor? If baby is fussy, could it be eg oversupply? Block nursing made a big difference to one of my babies, for instance.
IF the baby has a cow's milk protein allergy, that is a different matter. Lactose - free milk is a red herring and unrelated to CMP allergy. Lactose is just a sugar found in milk. Your body makes lactose in your milk even if there is none in your diet.
Kellymom on CMP allergy is helpful.
Whatever you decide to do, it may be helpful to keep a careful diary of your baby's symptoms now, to see if any patterns emerge. This becomes very important if you do start an elimination diet as it can be hard to look back objectively and be sure whether it's made a difference. If you think 2 weeks of eliminating CMP helps, you'd then need to challenge that really by reintroducing CMP for a few days, to check if things change or if, in fact, baby is the same. I think when you've done something difficult like changing your diet, there's a strong incentive to want to feel all that effort was justified and to want it to have made a difference. It is important to check, though, because if it's not CMP - yayyyy! Back on the ice creams for you!
Here's an NHS article on CMP allergy.