Sometimes in the afternoon I will put my son in his cot and close the curtains, put on some white noise and leave him. He often doesn't sleep but will instead just lie there playing with a teddy or quietly chatting to himself.
I use this time to have a brew or watch TV. Typically for around 40 mins - an hour before he stands up and starts whinging at which point I'll go and get him. Very occasionally he'll fall asleep and stay up there for longer but most of the time he just lies in his cot quietly.
A family member thinks this is mean if he's not actually asleep. Imo though if he's not crying then he's okay?
YABU - even if he's not crying, he shouldn't be left in his cot when not asleep.
YANBU - it's fine for you and he to have some downtime if he's not upset.