I make all the small talk/conversation with DF

[ShowMePotatoSalad]

And it makes me sad. He comes round to my house to see DS and he barely speaks to me. I ask him questions, how he is, and I get one or two word answers in response. He doesn't ask me how I am, how work is. He talks to my DS normally and makes loads of effort with him. I feel fed up of constantly doing all the talking. If I didn't try to make small talk would we just sit in silence? Would you do that in my situation?