The reason the government decided to fund free nursery places for all children is that research has shown that going to nursery is beneficial to their learning and social development.
It sounds as though the school believe that your child would particularly benefit from a nursery placement.
Most young children do actually benefit enormously from going, and it better prepares them for starting school at a later stage too .For your son not to go to nursery, may disadvantage him when he starts school, as most other children will have had the benefits from nursery.
It may be possible to fit your child going to nursery around similar hours to the school drop off and pick up times. Why don't you ask the nursery before making up your mind, if he can't go due the number of journeys you'd need to make in a day?
If your home is reasonably clean and safe, you won't be judged. Many people are in the same position as you with not being able to afford much. Co sleeping is purely your choice, and again is not uncommon.
You will probably be more judged (for different reasons) if you refuse the home visit to be honest, and it may raise welfare alarm bells.