I would simply erect a post and wire fence on the boundary OP. There's no need to move the current fence, and take no notice of those saying you were silly to put it where you did in the first place. You simply erected a fence on your own land, and hopefully left enough space on your neighbours side of your land, to maintain it. The fact that he's now removed the original fence, does NOT entitle him to use your land in any way, shape or form, so get out there, and at the very least knock in a couple of wooden stakes, to which you can attach 3 lengths of wire, one near the top of your posts, one near the bottom and one in the middle. Alternatively, if you have a little more cash, buy some cheap chain link fencing, the sort you would use for making a chicken hutch, and attach that to two posts one at either end, but please DO NOT delay this for a minute longer than absolutely necessary, as the longer you leave it, the more likely this CF is going to cost you money in solicitors fees trying to protect your own land. Also, don't take any nonsense from his builders, tell them that if they prop anything against your fence and cause damage, not only will they be paying for a replacement, but you will prosecute them for trespass as the land next to the fence is yours, and then show them where the real boundary is.
I went through something like this with a neighbour once, the boundary was ours, but the neighbour had come into some money, so asked if I'd mind if she replaced it with something a bit nicer. I told her I had no objection. The next thing I knew, her builders were in her words 'straightening the boundary', to her benefit!! I was absolutely shaking in my boots when I discovered what was going on, as I knew it meant conflict, but equally I knew that if I didn't stand up for myself it would be the beginning of constant piss taking. I therefore went out and told the builders that I had given permission for my fence to be removed and replaced by the neighbour, but NOT for the boundary to be tampered with in ANY way!! She came out and started getting stroppy, but I just stood my ground and kept repeating myself, so that it was clear I was not going to be bullied into giving in to what she wanted. I then kept watch until the fence had been completed, so that I knew it was going back up where it was meant to. Thinking back on it after the event, I remembered that she'd always had a bee in her bonnet about the fact that I could get my car right up the side of my house, but she couldn't, and she always insisted that that boundaries had been moved, but that simply wasn't the case.