So, to clarify, you've bought the land but haven't discussed the ROW with your neighbours, the planning department, a solicitor or the architect? 
I hope you're rich because this could cost you a lot of £££s.
We were in a similar position 3 years ago when we found a piece of land that we thought was perfect. It was situated between end-of-terrace houses and had been used for lock up garages.
We discovered that the two houses had a ROW, and not just a passage but a road wide enough to allow lorry access and all the way round the back of the houses. We should have walked away there and then, but we were naive and thought a cup of tea and a chat could sort it out. Well, 2 years later and a six figure sum poorer we learned some valuable lessons.
The first problem was the neighbours not wanting to give up the ROW. We persisted with the plans, believing we could win them round. The only place we could build was against the house, a party wall, but then we discovered the home owners had filed an injunction against any building until a full survey had been carried out on their wall and the effect our build would have on their foundations and house structure. This cost us a lot of money, and we would have had to pay even more to structurally secure THEIR wall before we even got near to building ours.
The next problem we ran in to was the planning department. The neighbour's house had a kitchen window at the back that prevented us building more than 1 story high. This meant plans had to be redesigned, and again, more money going to the architect.
The final problem we encountered was with the garages on the land. They all had asbestos roofing and had to be removed under licence by a specialist firm. And that wasn't cheap. More money down the drain!!!!!
I really, truly hope that you haven't bought the land already without doing all the necessary checks and end up like us – skint and distressed with a marriage under strain.
I wish someone had advised us of all this earlier, and I wish I'd never set eyes on Kevin Mcloud and Grand Designs.
And no, we don't live in our dream home on the land. After all the above, building took forever because the ROW had to be accessible at certain times in the day so the builders would just down tools and go to another job. A total nightmare – we ran out of money and will, sold the land at a huge loss and now rent because we're broke.
Good luck, but anything involving a ROW needs to be sorted long before any money changes hands 