Oh OP, it sounds like you're doing the best you can for your children and they'll see and appreciate that/you as they get older. I don't think it will make a difference as long as you provide a warm and loving home.
My step-children had a similar experience. Their mum 'left' them as small toddlers so my now husband gained custody of them. Their mum was bohemian, has never held down a proper job, always been in rented flats etc. She didn't really act as a mum to her kids when they were younger and was more interested in singing, drinking and having flat parties.
DH started seeing another woman and they, together, ended up earning a considerable salary and moved into a large 6 bed period home with many holidays abroad and the best of everything. The children eventually rebuilt a relationship with their mum, staying each weekend. From what the children have said (they're now in their 20s), they hated their mum being so flakey and unreliable when they were younger and also noticed a huge difference in the amount of toys/bedding/heating between houses etc. But, as they got older, they realised their mum was really cool, an old hippy and they love her to bits. One actually lives with his mum and she's still in the same flat.
DH was with the new, rich,, woman for around 7 years. They split up and DH walked away from the relationship with nothing. His wages was significantly lower than what their joint income had been and he moved into a small 3 bed rental. He couldn't really afford the rent and had to cycle 35 miles to work as couldn't pay petrol and he cooked the kids food grown in his parents garden. Again, the kids went through a big contrast as now mum and dad were living more similarly. One child struggled, the other accepted it.
DH and I met when the children were young teenagers. We, together, bought a beautiful house and our joint salary placed us as relatively high earners. The children, by this age, saw how hard we worked, how stressed we sometimes got with work, etc and gained a better idea of the hard work that goes on behind the scenes to maintain a lifestyle with a big house and holidays. They loved going to their mums as it was more chilled and relaxed. They are both in their mid 20s now and live much more like their mum than us.
What I'm saying op is a reliable parent, who shows love, warmth and encouragement to their children, is what a child wants and craves. I don't think you have anything to worry about.