the joy of mixed race relationships, hey!
They probably don't know much about your culture at all, they may have various misconceptions, they don't like your food, and I'd be willing to bet it all makes them feel highly uncomfortable! Indian weddings are so different, I expect they probably didn't enjoy it very much. I am white, dh is Indian, and my parents had huge misgivings about our relationship at first, simply based on so many misconceptions from lack of knowledge. Same as dh's family did of me, and it took many years to get past that.
I'd agree that to invite them over would be a good idea. You could do some fireworks, do the lights, buy sweets (maybe indian and english), and show them some of what your culture is about and give them the chance to appreciate it. If they still don't, sod it!
Also, I'd lay on a spread, and do non curry if they don't like spices, but I would draw the line at serving meat on Diwali. tbh, if it was me, I'd think sod it and stick at making it special for my immediate family.
How does your dh deal with it all? Does he support you in all of this?