We have a family friend of my parents who met a widower when she was also widowed. They'd know each other socially for years even when their spouses were alive.
They remarried each other in their early 50s (she was widowed late 40s).
She sold her house and moved into his (nicer house.) and I know she was unsettled by photos etc of his wife still on display.
I think you need to accept you can't erase the past.
I know this is different, but I had two long relationships before I met DH that stopped just short of us getting married.
The relationships didn't work out for various reasons. I'm still in touch with those men (DH knows) 30 years later. I still care about them very much. But DH is different and I love him for what he is, even though in some ways I had more in common with my former partners.
What I'm saying is that there isn't a Number One, but people we can love, in a different way, at different points of our lives.
It may be you are the rebound girlfriend for this man and he finds out he's not ready. I'd play is slowly and see how you feel, like any new date- widower or not.