Makes buying a normal ticket a waste of time
No. Buying a normal ticket allows you access to the theme park and to ride whatever you chose. With good planning you can still ride a lot of rides time. Its a darn site better than not being able to afford a ticket at all too!
If everyone bought a priority pass
Not everyone can. They are restricted in number. They can only sell a certain number per day. For those with time limits attached (such as the ones we have for Universal in Japan) there are only a certain number allowed per time slot. Likewise with the free Disney fast passes, which are available to all.
If nobody had them the queues wouldn't be shorter. All the people with priority would be in the standard queue - the queues would therefore be longer overall.
Those buying them rely on those who don't to gain any benefit
Obviously. 
There'd be no point if everyone had them after all.
But there is no point in queuing because people can just come along and take your turn
The queue times listed already take into account the number of express passes sold. So your queue time is unlikely to change because of them - not everyone with a priority pass will show up at the exact same time to use them. It may go up due to other factors such as a ride break down or an approaching thunder storm but not for priority.
You see the time before entering the queue and make your choice whether to join it or not. No one is coming to 'take your turn.'
A priority pass doesn't give you direct access to the ride. There is often a queue, just shorter. The people operating them allow a set number of people from each queue onto the ride. On some of the bigger rides with more than one track there are separate cars/tracks for standard and priority queues.