4years - I wouldn't worry too much about slippage. I did the same division of labour (shell under the control of the builer, self-managing everything else). As you already know, there are many stages of electrics, plumbing etc. so they will be prepared for repeat visits. Most of mine have been really accommodating and have dropped everything to come over to ensure the build hasn't had to stop - most of them live really locally, and this is a great advantage of that. The bills you will be paying them are large, and they will want to help.
Filo - Hmmmmm. I hate to be the bearer of negative advice, but I would have some doubts about working with a builder who hadn't worked with an architect before. The issue for me would be a simple one: can they read plans and spot mistakes? I know that sounds ridiculous, but the huge headaches I had with my electrician were due to the fact that, despite the fact that he was 65 and had been an electrician all his life, he couldn't work off a plan. He also wouldn't listen to me as I tried to reorientate him, or pointed out he'd got something wrong. In the end, I resorted to writing where things were to go onto the walls!! If this had been the shell, I think I would have been going even more spare - you do NOT want expensive mistakes with dimensions, as they may put you in contravention of planning! However, just because someone hasn't done something previously doesn't mean they can't do it. It just means they are untried at it! Attention to detail, care, precision, honesty and decency are often a matter of character.
I have had a bit of a showdown with the builder this morning. He put my oak upstand on the kitchen and steps yesterday and it was AWFUL. It was every combination of poorly put together and wonky. I cried when I saw it, and couldn't stop crying all night - all of the stress (and all of the tiredness of varnishing for 12 hours yesterday) just flooded out of me. However, while he might not be the most skilled joiner, he is a decent, honest guy who genuinely wants to please- so he has spent the entire day today sorting it out - and with much fettling, planing and sanding, it's looking very much better.