In a given moment, it's generally necessary to put the children first at that age - it's not reasonable to keep an infant waiting if they are in need of help!
I know a number of couples who would say that they prioritize each other over the children, and what they mean is that they make the health of their relationship a priority, even if it means some time away from the children, or slightly fewer activities or playdates for the children.
In practice this often seems to mean a 'date night' once a fortnight or month away from the children, a weekend away for two every year, children not sleeping in the parents' bed (beyond a certain age), children's bedtimes early enough to allow parents time together, limiting working hours even if it means fewer holidays or no private schooling (where parents can afford to not work maximum hours, of course) etc.
And - albeit only anecdotally - the marriages where the parents prioritize their relationship in this way seem to last and be happier, on the whole.
But these things are not always possible (for example, under extreme financial or health pressures), and there are many factors in relationship breakdown, so I am emphatically NOT suggesting that your divorce is because you didn't do this. Perhaps there's an element of truth in what your husband said, perhaps not, but if the marriage is over, there's no point in blame. My sympathy to you.