OP, your post rings a bell with me.
Especially your comment about you and your DH having "lived a lot" and being happy to spend time with your DD, whereas your BIL and his exP hadn't and so were happier to hand over your DN to your MIL. Are you me?
We're going through something similar with my DD and DN. PILs care for DN a lot whilst SIL and her DH continue to live a lifestyle not dissimilar to the one they had pre-DN. (DD and DN are both aged 2+.)
MIL has made it clear on several occasions that she wishes she saw as much of our DD as she does of DN.
In our case, my DH didn't exactly lay down the law with his parents, but luckily he has a great relationship with his dad (my FIL) and made it clear that our family time (DH, DD, me) is our priority, especially as my DH works stupidly long hours, often travels, and doesn't get to see us much.
We manage the situation by ensuring PILs get to see DD on a regular basis on our terms (monthly, as do my own DPs, and we plan a couple of months ahead) and every 3-6 months or so we ask them to babysit in some capacity so that we can have a night out or a night away. This is a drop in the ocean compared to what they do for DN.
However, because PILs know DN much better than our DD, and because they are the same age, when they look after DD they forget her preferences (food, toys, books, TV, etc.) and often just foist on her what they would do with DN (or what SIL dictates they should do with DN). Which is a real shame now that DD has the ability to express her own thoughts, preferences and feelings.
I try to look on the good side of this. I have PILs who are willing and able (arguably more willing/able than my own DPs) to look after DD whenever I ask. With DC2 on its way soon, that's definitely a positive!
The flip side, FWIW, is that DN sees PILs as 'primary carers' and plays them up at every turn, just as she would her own parents. PILs don't seem to enjoy spending time with DN - it's a chore not a pleasure.
Let your DH do the talking, as you suggest. If he feels the same as you, which it sounds like he does, then that will be a good basis for moving forward.
Good luck!