The people across the road from me have done exactly the same, also on a sloping site.
A former colleague lives in the house that backs on to them. They were livid, as the slope means that the "garden office" rises several feet above his fence and shades his garden. They contacted the council, who said that it didn't need planning permission, partly because it had no cooking facilities and therefore couldn't be used as a dwelling, and that height wasn't an issue because the building was within the permitted height at the front.
However, it does have a toilet and shower room. Within weeks of it being finished, it was listed on airbnb, with a kettle, toaster and microwave clearly visible in the pictures. Former colleague was livid, and got on to the council, who said it still doesn't count as a dwelling because the microwave etc doesn't constitute "cooking facilities".
For years, it was used regularly by airbnbers, who often seemed to rock up in two cars in a street where parking is very limited. On more than one occasion, I had people knock on my door, sometimes very late at night, because they've mislaid the number for the keysafe and can't get in, and the owners were away. Thankfully, they rarely have guests these days and it's no longer listed on airbnb.
They managed to piss off loads of people at our end of the road though, especially as they've now got planning permission to extend the house sideways which will near double the size of it.
I don't think YABU, OP, but sadly I doubt if there's anything you can do about it.