Yes, this is very odd behaviour.
Let me get what I think the story is so far so that my advice makes sense.
.
.
.
You recently moved to a small village with your DH and (at least) 2 DCs.
You do not know many people in the village that well yet.
A man from the neighbourhood (not your NDN) often stops to talk to your DD.
He has started letting himself into the garden.
When you posted this, he had let himself in and got on the trampoline with your daughter (who is only five).
You were inside tending to the baby at the time.
You called your daughter in (not sure on this point)
He left
You did not confront him.
.
.
.
So far, my thoughts, (and please correct my version of events if it is wrong). He is behaving inappropriately. That much is obvious.
How I think you should proceed:
Have a friendly chat with your DD about stranger danger. Just to refresh her memory. Remind her not to play with people she doesn't know, not to invite people over without permission, to come and get you if she feels uncomfortable. Make sure she definitely knows not to leave the garden if a neighbour invites her around to their place.
Later, maybe ask her about the neighbour, e.g. what's his name, what's his job, what was their conversation about. Keep it all very friendly but try to find out what is was about.
Chances are, this was a friendly neighbour. However, his behaviour has crossed several lines.
I wouldn't confront him unless he does it again. Next time your DD goes out to play, if possible sit outside with her and the baby. If the man comes across, have a chat across the fence. If he tries to invite himself in, tell him that you'd prefer neighbours not to enter your garden without an explicit invite and that your DD needs permission to talk to people. No need for further reasons, imo.
If he doesn't come over when you're around, but comes over every time your DD is alone, something is very fishy. Maybe not in a harmful pedophile way, but fishy nonetheless.
If he comes over again whilst you are inside, particularly if it is after you have confronted him, you will need to march out there are straight up tell him that you find his behaviour inappropriate and would he please stop. You do not need to justify asking an old man to leave your garden and stop playing with your young daughter, particularly on a trampoline!
Does he do this with any other neighbourhood kids? I find this all very strange.
BTW, I find the Sarah's Law preposition very reasonable. Chances are he doesn't have any past convictions but you would be kicking yourself if you found out he had and something happened. There is a very small chance he is grooming your child. Just make sure it doesn't escalate to the point where he can convince your DD to leave the garden with him and he can reasonable argue that he didn't know it was wrong because you have never said anything.
All that said, I also hate confrontation but when it's your child, you have to do something.
Good luck with this.