Legally it belongs to his children. In practice if you quickly transferred the money it seems unlikely anyone would come after you.
There's already lots of very good advice on this thread, but it would be astonishing if somebody didn't come after you. Even if you transferred the lot a day/few days before, it would still look very suspicious.
Assuming that you have no funeral plan in place, his adult next of kin (not you, I'm afraid) would be responsible for both organising and paying for the funeral. The very first thing they would surely ask is
whether he left any money that could go towards it - and then they'd discover that you had taken it. Legally, accounts have to be frozen at the point of death (obviously usually applied retrospectively, as nobody calls the bank five minutes after their loved one dies).
You need to realise that, in the scenario that he only has a couple of thousand in his account and dies, that money would instantly be gone and needed to pay his funeral. The question shouldn't be 'am I OK to take the £2K?' but rather 'how would the extra £2-4K be found on top of that to pay for his funeral?'. As you aren't married, the burden to pay any excess would, I suppose, fall to his adult next of kin (parents/siblings etc), but you wouldn't then legally have a say in his funeral or burial/cremation arrangements or anything - also no spousal death benefits from the government.
There are some horror stories, as recounted on this thread, where partners are instantly shut out when a person dies tragically young, and parents take over; however, at the point of marriage, that's when the commitment and legal declaration are made that you now take responsibility for each other and sever links with your parents/family as the most important other (adult) person in your life. You take on joint responsibilities at the same time as joint rights. To be fair, just as the government has had no indication from you of your wishes for the above to be the case, how do your late partner's family know just how seriously committed you were?
The aspect of love in a marriage is entirely personal, completely separate and of no consequence whatsoever in law. It is usually the reason for deciding to marry, but in no way indicative of the actual marriage itself.
Legally, you mustn't think of a marriage as a way of expressing your undying love for each other, but as a way of saying to the government "We hereby share these joint rights and responsibilities between us and, when one of us dies, we automatically nominate the surviving spouse to have full control over every 'next of kin' decision."
By default, by not marrying, as far as the law knows or cares, you are effectively stating that you do not want all of this as, if you did, you would have demonstrated this in the accepted legal way by marrying (or CPing).
Unless/until you marry and make a will, the very first thing you should do is set up a joint account and both transfer all/most of your money into it (assuming you consider it all joint money). In a very limited sense, a joint account is legally like a snapshot of a small financial part of a marriage, insofar as the bank and government would expect you after the death to take sole control of all of the joint funds/spend/transfer them to your own account etc. - and not be suspicious of you in any way.