I don't think you really 'get' what passive aggressive means if you don't think you were being earlier by not hinting that it was your anniversary, and waiting to use it against him later.
Yes, if it means a lot to you, shame he did not remember, but fgs, if you don't hint a little then it can happen. You have not given him the opportunity to make it up to you. If you did, I am sure you'd feel so much better and not so miserable, all alone at home while he is out having fun.
I am married, we do not really celebrate things like anniversaries, or Valentine's, Mother's Day/Father's Days go unmarked by cards or flowers.
However, I get coffee in bed in the morning, lovely flowers for random things, he takes the kids out lots and lets me lie in the bath for a read and some peace and quiet, I also give him time on his own.... etc etc.... and we spend every evening eating a good homemade dinner at the table, talking to each other and having a relationship!!
Now, as far as I am concerned, that is what matters. Not all these games of one-upmanship and guilt tripping.