Hullo! I have been a judge for a similar competition (though, in our case all the books were mostly traditionally published already).
In our case we tried to have at least two people read the first 10% of all of them. This would be realistic for children's books (as they're shorter), but to be honest, you could probably tell before 10% with most.
Initially I tried to read more than that, but was going insane until another judge pointed out that if the beginning wasn't catchy enough it ultimately wouldn't make our long list anyway. I felt guilt free after that.
It wasn't so much that most of the books were dreadful, it's more that the few that were exceptional stood out. You know when you see something really good and flag it up for others.
We long-listed, and drew each other's attention towards good ones/ ones we were uncertain about- so we definitely shared the decisions.
We had around two hundred books and it took us three months to collectively rule out 150. For a few (the self published ones, mainly) it was spelling and syntax- tense confusion often. But it was also often topic- ideas that didn't feel fresh, titillating female violence/unnecessary sex (when you read a lot of books in one go these two features feel particularly annoying), unconvincing voice.
In about ten cases we ruled out a book in under five minutes (three of them were for characters enjoying being raped!).
We obviously questioned out decisions often, but the thing that I found most interesting was that two other major literary prizes in the same year (the writing world is small. We had contacts!) ended up with very similar long lists to us, or had books on their long lists which we'd discussed heavily. It showed me that recognition of quality it much more universal than I'd thought- even if the criteria you're using feel intangible!
I read everywhere that year. I spent about three hours a day reading, so I really didn't watch tv. I read in the playground watching my kids, I read on the bus, I read while cooking tea. In the end I realised this was s really good test. A book that could keep my attention in those circumstances was worth revisiting.