I'm going to give advice, hope that's okay!
Firstly, open the channels of communication. Tell him that you find it very hard to work when he comes downstairs, as his breaks aren't the same time as your breaks.
Sit down with him and make a timetable. Yes, I know that's giving you more mental load, but he is obviously oblivious to what's happening. In your timetable, put what number of hours each of you works, when you have to do pick up of child/ren and who will do that. Schedule joint breaks - lunch or a quick ten minute teabreak together.
Have core hours when you cannot be disturbed. Make sure he understands he can't just keep chatting to you just because he's in the same room.
Also put into this timetable all non-work hours, and cleaning times for both of you. You can both have this as a flexible timetable - update it weekly, or whenever you need to.
I also think you need to separate your working space from family space. If you have to work when the toddler is at home, set up an area in your room so they can play when you're working. If your toddler is at home when your dh is working, I really do agree with those who say a stairgate, and yes, if you're the one looking after the toddler, you'll have to make sure they don't barge in on the office (even if it means getting a lock for the office door). He must give you the same deal if he's in charge of toddler.
Separating your workspace - it sounds like your dining room is in the same space as your kitchen, so I think you should get one of thosefolding changing screens and a little desk (even if it's a foldaway one) and set it up at the far end of the dining room. If you need to, you can put a "do not disturb" sign on the outside when you absolutely cannot be interrupted. That way, you've got your own office, too, and he can't confuse it with you being "at home". And you can fold it all away when you need the dining room to be family space.
I honestly think that he's not going to offer up a solution because he just doesn't see you as "at work" because you are in a family space.
I used to work in a big open office, and there were always colleagues who needed to talk or interact with others all the way through the day, especially when work was boring, so I can see why he's doing it, even though it's driving you batty.
Hope some of this is useful