OuiOui, this sounds really hard for you, and to be honest it's not good for your ds either. You both need your sleep. And this can't go on forever. First of all, do you think there may be any kind of illness that's affecting your ds seeing that he's also gone off his milk at the same time? I'd rule that out before doing anything else if I were you.
My dd was the opposite to your ds in that she was a fantastic sleeper from about 2 months old until she was around 15 months then a period of illness / some anxieties messed it all up and she was really all over the place and as you describe, not even predictably bad! We tried everything, I mean everything, before we resorted to cc and by the time we tried it we were desperate and had very good advice and read Richard Ferber's book so did it by the letter and it worked when all else failed. It was horrible for 2 nights, then not too bad for 3 nights and at the end of 2 weeks she was sleeping like a dream all night with no problems at bed time. She's 20 months now and still sleeping very well. More to the point if she now becomes unsettled in the night we know what to do to reassure her without causing another sleep problem to arise.
I totally respect your decision not to do cc but I'm wondering why you are so against it? You may have a completely valid reason, but I just wanted to say a few things in its favour. Before I read Richard Ferber I think I thought it was a cruel technique in which you left a child to cry but in fact it's not really like that and you can actually do it in a very gentle way. Even if you absolutely don't want to do it I do recommend that you read the chapter of his book called something like 'what we associate with falling asleep.' It made so much sense to me and dp that it encouraged us to try his method. But you have to believe in it and you have to do it properly.
Just to recap slightly what Ferber says about sleep associations. He asks the reader to imagine how they fall asleep themselves. We have a bed and a pillow (usually). Now if someone took our pillow away in the middle of the night and we woke up and found it not there we'd wake up with quite a shock and be frantically looking round for it. If this started to happen regularly every night we'd also start to be anxious about going to sleep, wondering where / when our pillow disappeared. Ferber believes that all the 'putting to sleep' techniques we parents to, be they rocking, feeding, patting the back, are relied upon like we rely on a pillow. They are what a child NEEDS in order to fall asleep. However, every time they wake up and their 'pillow' isn't there it makes them anxious and they wake fully until the 'pillow' returns and they can sleep again. Does this make sense?
In actual fact, all the 'kind' things one does to help a child to sleep are in the end not particularly kind. We, altough never ever intending it, are often the cause of our child's sleep problems. They may prevent crying in the immediate short term but they don't make for a confident sleeper and instead of going through a very short period of crying when doing cc, you are prolonging the bad sleeping for months, perhaps years, perhaps a lifetime. I was a child who was 'put to sleep' by my parents and I've been an awful sleeper all my life, very insomniac.
Ok, this is just one opinion. You'll get lots of others which may be much better. Let us know how you get on?