I was like you & lost weight & back in my usual jeans withing days of giving birth, & then lost more, & strangely I WAS eating diary, despite normally being intolerant to it,
I had such a hard time with food in my early pregnancy I was very worried about my nutritional intake & tried drinking milk, weirdly my intolerance had gone completely, & stayed away until 3 months after I finished BF at 10 months, whilst BF I craved milk, drank up to 10 pints a day, & still didn't put any weight on
Ar you sure you need to give it up for babies sake??, my DD is also Dairy intolerant, & my drinking milk had no effect on her whilst BF, unlike eating anything with additives, or anything from the onion family, which all had a big effect, either on her Skin, or colic....thankfully I learned that one pretty early.
Also it might be worth you speaking with your GP, & getting your Thyroid checked, have a look at Dr Peatfields, "Your Thyroid & How To Keep It Healthy" .....over active Thyroid can cause you to loose weight like this,
I was always very slim despite eating whatever, lost weight like you, & then after finishing breastfeeding became ill, & put on 2 sone in weight, thankfully I was slim enough before for it not to be too much of a problem size wise, I've now tested positive for Hypothyroid..... UNDER active, & they reckon I was actually OVER active before, & its common it switches over, & Hormones play a part......
You might want to look into Spirulina as a supplement, it wont help you put weight on, but will help make sure you have a good nutritional intake
Good luck