I'm assuming by "currently not that good" you mean you are in a bit of a relationship rut rather than that's there's anything fundamentally wrong with your relationship, OP? (Or indeed any abuse?)
If it's just that you've been slogging away at day to day life (which can be really hard and draining and seemingly endless, especially when you have young kids) then everything everyone has suggested above will help - but I guess the big question is how to get started. By which I mean, how to open up a conversation with your husband. (By all means, I'm sure if you singular started doing some of these things they would help, but it would be a lot better if you were both pulling in the same direction, so to speak!)
So... Do you feel you could pretty much tell your husband what you've written here, that you feel you're both really good at creating and participating in great quality time with the whole family but now the kids are a little older you would like to invest a bit more time and energy in the two of you as a couple again as well?
I'm sure you've seen it recommended on here before but The Five Love Languages is a good book (basic premise: people give and experience love in different ways which from memory are quality time, physical touch, words of affirmation, gift giving and acts of service - people tend to naturally offer love in the way they prefer to receive it - if you have different primary love languages this may cause one person to feel unloved even though the other person does love them and believes they are showing them love - this is easily solved by learning your partner's love languages and doing more of those things for them - I have oversimplified this and I'm sure there could be more than five languages but you get the general idea of what the book is saying), if you're looking for something specifically related to improving your relationship to do together then you could (separately) read a chapter a week and discuss it over a meal together?
Or a few years ago my husband and I did The Marriage Course which is run by lots of churches (it's not a Christian thing, it's one of their community things like running baby groups or lunch clubs for the elderly). Each week there is a topic like differing communication styles, how to resolve conflict, money, parents and in laws and so on. We weren't experiencing any problems in our relationship, we were just thinking about trying for a baby and thought it would be a good time to do something like that. There were around ten couples there but there is no group stuff, you just sit in your couple (it was quite sweet actually, they set up small tables in the church with tablecloths and candles and everything and did a meal so every week it felt like a proper date). I think quite a lot of people go who are experiencing more difficulties in their relationships so if you're wanting something more structured then that might be a good option.
But I think the important thing is to open up the idea with your husband, because if he feels the same as you there are loads of really simple things you can do to improve your relationship (we always have a television programme on the go that we are watching together, we go out for lunch dates when the grandparents visit and can have the baby etc.).
Im sorry this has ended up like War and Peace! (Which does mainly seem to be about relationships...)