Ok, having read the rest of the thread which appeared while I was typing my thoughts are that because this has happened very spur of the moment there’s been very little time to put any thought into this on either side and it doesn’t necessarily make either party right or wrong but some further thought and discussion does need to be had IMO.
Firstly, from your perspective, your aunt has literally just been taken ill and the prognosis is presumably unknown at this stage, which means that while practically you’ve decided to travel down there immediately in order to help out your mum, this isn’t a long-term solution but could become one from your dp’s perspective if either the outcome remains uncertain i.e. she stays in hospital, or if God forbid she dies and permanent arrangements need to be made with regards to care of the children who it seems do not have any immediate family beyond your aunt. But all you can think of is the here and now....
From your partner’s perspective, IMO you taking the baby away is a red herring. From his perspective a young-ish family member with no other immediate family and three young children has been taken seriously ill with the prospect of not surviving, and his partner, i.e. you, has immediately rushed out to be with those children in order to support them through this as yet unknown time. It’s entirely possible that your DPs thinking is on a different level to just that of not seeing his baby, but may be even subconsciously on the level of this aunt not pulling through and you potentially offering or be expected to take on the care of three young children. Something which he certainly won’t have bargained for and neither will you have presumably, but given the seriousness of her illness some serious questions will need to be asked in terms of what is going to happen to them, as presumably your mum isn’t in a position to take on such young children herself given her own children are now adults. Where are the fathers of these children? You say the one father is out of the picture and has been for five years or so but what of the two year old? Presumably they have a different father and immediate family to the older two?
I do think that you should go to support your mum in the immediate aftermath, but given how quickly you have dropped everything to support these children who potentially are going to have no mother in the long term I don’t think that your DP would be unreasonable to be a bit concerned by the fact that A, you dropped everything with such short notice and no discussion with him, and B, the possibility that this will become a long-term issue if the aunt doesn’t pull through.
I would go, but I do think that discussion between you is needed in order to resolve the real issues arising from this situation. Because I actually don’t think this is about the baby, but that the baby has been able to be brought up as a reason in the immediate, given that the actual potential concern isn’t something one could mention so soon without looking completely heartless even if it is very relevant.