No, there's usually no way of telling, as you say. Unless you very clearly remember an incident for some reason, or have witnesses, and his description of it is quite different. Even then there's no way to be certain whether he is deliberately distorting things or just misremembers.
What you propose to say is an excellent response (some might suggest you leave out the "sorry" or change it to "Really sorry if you were upset about..." - . It will only make any difference, though, if he's on the level.
My relationship... it would be a major saga. There are so many aspects to it. It's amazing how often I read threads on MN and go "Hey, XH used to do/say that exact thing". So what's relevant, let's see...
I believe in communication, or to be more accurate I'm a hopeless motormouth, whereas he's more what you might call the strong silent type or, to be harsh, a moody bugger. I would always try to talk things over, explain, clarify. He'd assume I knew what he was thinking and had done something different on purpose. The first I'd know I'd done something wrong was the bear-with-sore-head impression, and it would take several days of digging to get him to tell me what it was all about (much of that playground nonsense like "you know, of course you know" when obviously if I'd known I'd have either apologised for it or explained it). Eventually I'd dissolve into tears - or in later years when I'd had just about enough of it, get quite angry. Then it would come out, and it would be some really stupid thing. Like one time we'd been watching Mastermind and one contestant's specialist subject was the football world cup. XH was quite scathing on this as a subject, and I agreed but added that I thought it was quite clever how the researchers managed to make challenging questions on it. Just then one of the DCs started a conversation on the side with me (happened a lot! no quiet telly time in our house!) about our various roles in the family. I said something like well your daddy is extremely good at fixing things. A bit later he had That Expression on and did the silent treatment for two days. I eventually dug out of him that he assumed I fancied the bloke on Mastermind and thought he was terribly clever for answering questions on stupid football, whereas all he (XH) was good for was fixing things - the latter said in a high whine that was supposed to be an impression of me. I explained that it was two quite different conversations and I certainly didn't fancy the Mastermind bloke one bit, and that I was extremely admiring of his fixin' ability, but he didn't seem mollified. I said look, we really aren't communicating here, I think we need a course of couples counselling. He said in wolfish tones "This marriage is over the moment you set foot in one of those places". I said "It's over if we don't". I must have looked as if I really meant it, because next thing he was terribly sorry and I should have realised he was joking and we could sort everything out between us and I was the love of his etc. The next day it felt like we'd come to a new understanding, marriage revitalised; he said he would never have a go at me like that again, took me on a rare shopping trip and offered to buy me an eternity ring. It was overpriced and would have come off my credit card anyway, so I passed on that.
Or the sex thing. Any time he was up for it but I wasn't, I'd be terribly polite and apologetic but he'd still sulk for days. He would call my period "the convenient time", because he said it gave me an excuse not to be available . One time when I said not tonight please, I'm awfully tired, he told me that that was it, he was never initiating it again and if I ever wanted it I'd have to take the lead. This appalled me, as I'm sure was the intention, as for one thing I'm not that great at rejection either, and I've never been very confident in the bedroom department (he was my first and only lover, I wonder if that had anything to do with it?). Fortunately, or in hindsight unfortunately, he didn't mean it.
I used to work up in town. He'd drop me at the station in the morning and pick me up in the evening so he would have use of the car. In fact even when we lived 10 minutes walk from the station he'd insist on doing this, even though I was looking forward to the extra bit of exercise and not dragging him out with the DCs at ungodly hours. He said the area was dangerous. (It really wasn't.) When we moved further away we got another car, but somehow he still nearly always needed to drive me (he'd be doing some major works on one car which would somehow drag out for weeks, or it would have some mysterious ailment that he couldn't diagnose but which would be ever so unwise for me to drive). And he'd start off from home ridiculously early to meet me from the station. I'd call home to say the trains were up the spout and I was going to be late, or could he meet me at a different station; the DCs would say it's too late, he's gone already. Or sometimes he'd bring the tribe along, and I'd finally limp off the train all knackered to find WW3 taking place in the back of the car, and XH sitting in the front with That Expression on again. When I forced a mobile phone on him, he forgot to switch it on. So I got the DCs mobile phones too, so he stopped taking them with him to the station. Sometimes my train would be cancelled. He would say that it couldn't have been, because one came in at the same time it always does and the same people got off it, except, of course, me. I wwas bewildered by this, as obviously I'd been at the station the other end and the train was ever so cancelled, and the usual bods were all grumbling with me on the platform! I wondered if he'd been confused by a train coming in from a different line. It was only after we were divorced that it suddenly struck me that in fact it might not have been true at all. He had no problem with telling bare-faced lies on occasion (eg the smoking issue, which I've grumbled about at length more than once on MN; suffice it to say I never told him he had to give up, and explained till I was blue in the face that honesty was a lot more important to me than not smoking, but he would still swear he had given up (again) even when there was a still-smoking stub by his foot and he reeked of it. The stub was "old" and the smell was my imagination).
OK, these small snapshots are a saga already, but there's a few more... When we'd been out to friends, he'd hiss at me afterwards "Do you know what you said to Aunt X?" I thought we were getting on fine, she didn't take any offence as far as I could tell. But apparently I'd said something truly awful and insulting. Actually once or twice I have said something awful or insulting, purely by mistake, and was devastated when it was pointed out to me afterwards. I suppose he must have found this entertaining because he'd make up things I was supposed to have said, or put a very strange twist on what I did say. Say, the petite woman who asked me to reach something down from a high shelf in the supermarket, saying self-deprecatingly what a pain it was being short; I complied, then indicated my very pregnant tum and said I might need her help getting things off the bottom shelf. "Do you know what you said...". Well I can only say, she didn't appear to take it as an insult. But she was only being polite. Of course. Then again, maybe it was an awful thing to say...
There were the aches and pains. Every winter he'd be afflicted with terrible discomfort. It was unbearable and one day he'd "go off a high place" (ie end it all), unless it turned out to be a fatal illness which would save him the trouble. I kept saying go to the doctor, it's probably rheumatism, it isn't life-threatening but there may be something that can help you live with it. No, no, I should just make sure he was well insured. After I'd told him we were definitely divorcing he went to the GP for a check-up, and came back all full of himself. Said several times in a row "I'm fit as a fiddle". What about the aches and pains, I asked? "What aches and pains?" he said. I said the ones you get every winter. He didn't know what I was talking about.
And don't get me started on infidelity. Just... don't. I thought for years that he was insecure. It only occurred to me when we were well on in the divorce process that he probably knew bloody well I hadn't done anything of the sort. He just knew I would launch into defending myself, explaining away the incidents he thought looked like flirting, and assuring him (as I bet my last breath he knows damned well to this day) that I am an honest person, and if I fancied someone I would tell him. All the times I cried and pleaded and assured, until eventually he kindly forgave me for something I hadn't even had the pleasure of doing! The last straw was when he told the DCs that he had heard me arranging to meet a man I'd met over the internet. What he had heard, in fact, was me chatting to a group of friends with whom I was playing an online computer game, about the game. Absolutely nothing that could be construed as arranging a meeting, however you twisted it. But the DCs didn't know who to believe, and the youngest one (9) cried.
Oh, there was that time I said we seriously had to talk about mortgaging the house. He said look, I've bought you some birdseed, and waved it in my face. I just stared at him helplessly, then went and locked myself in my room and howled for hours. I vaguely considered suicide, but we'd run out of pills. (It was the wrong sort of birdseed too.)
Long? Oh, but there's more, much, much more... some 28 years' worth of more. A (very strange) year dating, 23 years trying to make the marriage work, another 2 living together while divorcing, and the odd few things, or the few odd things, he's done in the couple of years since. One of several very liberating moments was the day I realised that it didn't matter any more what he believed, or claimed to believe. Nor did it matter whether he was doing any of this on purpose, though it would be interesting to know. I don't have to live with him any more. It may take a few more years to reclaim my sanity, but I expect to get there eventually. I twisted my brain every which way, trying to make sense of what he said, when all the time what he said in fact made no sense. Trying to understand why he felt that way about something, when either he didn't really feel it or a sane person would not. Now I have to untwist my brain and re-engage with Planet Earth.
Anyone still awake? Have I crashed the website? Did any of that make any sense? I'm not surprised if it doesn't. It doesn't to me, and I was living in it.