One classic always suggested on MN is to answer the door with your coat on or with your dc's coat in your hand, getting pram out etc so that you can say sorry, it's not convenient, you're just getting sorted to get yourselves out and to the shops/baby group/ appointment (ie you need to go now as you're already running late...) or similar.
You might need to follow it through a few times or have a ready excuse if you don't go out straight away and he's watching (got a phone call from my mum and she really needed a chat, went on a lot longer than I was expecting, I must make more time to call her during the day...)
Or answer the door with your phone in your hand as if you're in the middle of a call - say hang on to your caller, say sorry, you can't talk to neighbour as you need to talk to caller, and you have a busy few days coming up but you're sure you'll see him after the weekend/in a few days...
But as others say - only open the door a little bit and stand there so that he can't physically get past you. It feels rude to start with but that's just conditioning - most women would never dream of walking into their neighbour's house without being invited in, particularly in these sort of circumstances. But stop and think about it - it's rude of him to walk in too - even more so - without being invited. So next time, take a deep breath and stand there - ti's easier if you have the phone in your hand and you're talking (allegedly 
) because you can hang on the door on the one side, rest your other side and phone/ar,m/hand against the door frame and look away from him to the phone and talk to the phone so there's not an awkward silence while he is looking at you expecting you to automatically step aside and let him in, meanwhile you're talking to your 'friend' and saying I'm just talking to Old Bob the neighbour, he's just popped around but don't worry, we hadn't arranged it and I've promised you I'll talk this through to the end (relationship, CV, annoying neighbour problemns - whatever springs to mind if neighbour presses) and then say no, it's really not a problem, Bob understands that I have things that I had already arranged to do, he can't always turn up and expect me to be free for a long chat... so you're saying what you want him to hear without having to say it directly to him if it's difficult - and then you get to turn to him and say I'm sure y ou just heard, I really need to take this call, I'll catch up with you in a few days as I'm really busy tat the moment! Bye! And just stand there until he starts to move - if not, talk to your friend again and say that bob's just about to go, you'll be back able to concentrate on her woes in a moment... And then if bob is still there, point out that you really really need to go and carry on your call, you'll pop round when you're free and then slip out of the door and shut it as efficiently as possible tso that he can't get a foot in!
Also - Is there any chance that he has got completely the wrong idea and thinks that you might like him a bit more than him being just a random old neighbour? It's happened on more than one occasion as politeness is misinterpreted as a come on... Scary but one to be nipped in the bud if so!