I wouldn't do it and I don't get on badly with my ILs. Certainly wouldn't do it with my own parents. It's the kind of situation that can be good when all's well but has the potential to be hideous if it starts to go wrong. If you DH is not keen, that would be warning enough for me!
We lived next door to my Mum and Dad for many years til my Dad died and my Mum sold up and moved permanently into their home in Spain. It was fantastic and I miss it incredibly. We had very clear guidlines in place which meant we all kept our privacy. Was fantastic for the children who would wander between the two houses, but always maintained our own lives. Friends thought it meant we had a babysitting service permanently on hand but this wasn't true. Probably relied more on MIL who lives 100 miles away much of the time because my parents had a very active life and were often abroad or doing their own things. Only down side were the arguments you couldn't keep quiet would be interupted by my Mum on the phone reminding me they could hear much of it
If your partner isn't keen then don't go. I love my dad but wouldn't want to live next door to him. You are never really allowed to be an adult with your parents. I'd rather have less physical space and more emotional freedom than the other way round.