Why don't people RSVP even if you specifically say let me know by XX or RSVP by XX?
It's more awkward if you don't give a date but surely people realise you need to know a couple of weeks before (especially if invitation was sent 6-8 weeks before)
Most people seem to be considerate and respond in a timely manner then you have the odd few which makes it hard to plan fully.