Hi Jo, after reading lots of threads on mn about sleep the key things seem to be getting them to settle and resettle by themselves, little or no attention, as mentioned be very boring, no eye contact, no fumbling from one thing to the next cos even though you're not 'playing' with him, all the visits and trying one thing or another is attention still the same.
Now I'm no expert, but we learnt the hard way with ds and applied the things we learnt with dd and she's been a brilliant sleeper (touch wood!). With ds we used to tuck him in tight, hold his hand, rock, hold both hands, feed, sing & rock, pickup/putdown and nothing worked (we tried all these things for a number of weeks at a time). Eventually enough was enough and we left him to work it out for himself with me visiting every 5mins, no eye contact, check he's ok and out the door again. Sometimes he'd play, sometimes he'd cry (only once or twice really hard for about 10mins solid), but in the end (over about three nights we did this) he learnt to settle by himself (no boob) and resettle in the middle of the night. He only ever used to sleep for 45mins stretches in the day and the resettling at night, amazingly, helped him resettle in the day and he moved to having 1.5hr nap/s.
With dd, once she was on solids and I knew getting enough milk/food in the day, this was around 5.5/6mo, we left her to resettle at night and she grizzled for about 10mins, 10mins hard, 10mins grizzle, then off to sleep. It only took the one night. Whenever she's woken in the night since I've put it down to teeth, gave her some bonjella & a cuddle, walked out and she'd go off to sleep within a few mins.
Now, I don't like the idea of leaving them to it myself, so I went in every 3-5mins to check on them, sometimes would pickup, cuddle for a couple of mins or until calm, and would put them back in say 'it's time to sleep', or say nothing at all, and walk out. No eye contact is sooooo important. I knew they were safe, they hadn't hurt themselves, knew they were tired and I knew they would go to sleep at some point (Twiglett told me this, they're in the safest place they could be[in their cot]). DD has a muslin to cuddle, ds never took to one. Neither had a dummy so can't comment on that. There are lots of small/medium soft toys in there to 'buffer' the edges (had to take out bumper around 7mo). Both of mine are/were in Grobags.
Whatever you and dh decide to try I think you have to stick to the plan for atleast 2-3 nights. It takes that long to break a habit. I also think your ds is caught in that over-tired circle and things won't improve until he gets more sleep. They (meaning mine) always have a couple of bad days if they miss out on sleep and it takes a bit of effort to get them back on track. I suggest you agree on a plan of action and try the first night when you have a few days when you're not going out etc. and you can concentrate on putting him to bed and down for naps before he gets over-tired. The first night is always the hardest.
Sorry it's so long. Will follow this thread to see your updates.