You need to get a routine in place and stick to it regardless.
This. Your DH is being the problem. He is acting like this is OK and that is damaging your ability to get a routine in place and stick to it.
Think about "sleep hygiene" and the use of lights in your house.
You need to have a relaxing evening routine where big brightly lit screens are switched OFF, there is a bath or something like that to calm the baby down, the lights are dimmed, gentle soothing music in the background and so on. Make sure your baby has a place to sleep that is not a place where they hang out and play during the daytime. Do a book, lullabies, whatever.
In the morning, set an alarm and get everyone up-- snap the lights on, draw the curtains, get some fresh air into the house, and make sure you get baby outside for lots of daylight and fresh air to re-set his natural body rhythm.
It sounds like you need to get grandma on side. I am very surprised that she is not putting her foot down about her grandchild being kept up at all hours? Is she lonely and craving loads of time with her grandchild? It's understandable, but the baby's needs have got to come first. If she won't get on side, you need to be very firm and tell her that "bedtime is at XX:XX which is in an hour's time, so we are about to start helping baby to wind down and get ready for sleep. I think he will settle better if it is just us, as he is always so excited to play with you whenever he sees you. See you tomorrow!" As a PP suggested, you can always use your doctor or health visitor as the bad guy ("The doctor said that we have to....") if you are worried about causing hurt feelings.