I could never think of books as clutter - it worries me that people could. To me clutter is all that stuff that you can't find a use for - the lamp stand that just needs a new shade - the soda stream you might get out one day - the bag of clothes that don't fit anymore.
Books aren't clutter, because they aren't useless, or purposeless, unless you know in your heart of hearts that you're never going to read them again, use them for reference, or wish to pass them on individually at the right time.
I know some older, very bookish, people who got rid of their life's collection of books, and it was actually very sad, as if they were preparing for death, or turning their back on life, or simply declaring that they didn't need their education anymore.
It also depends on the books. If I've bought something for light reading and wasn't impressed, and don't really want it around, then I would get rid of it to a charity shop or amazon, as soon as I was finished with it. But there are collections of books I will never get rid of, that I will gladly move from place to place, because they mean a lot to me.
If the books are just stuff to you, then sure, get rid of them. Though I
I know some proper hoarders, and can honestly say that having a lot of stuff you like, or that means a lot to you, is not the same as hoarding.
If space is becoming a real issue, then I would do an inventory and divide your books into three lots. Those that you definitely want to keep, those you can easily get rid of, and those in-between. Get rid of the one's you won't miss and see how you feel about the rest. It doesn't have to be all or nothing.