At risk of being controversial here, I wouldn't worry too much about not having 100s of followers - as far as I'm aware, being a follower means you get each new post emailed to you, but that doesn't guarantee that the followers read them! I follow about half a dozen blogs that I really like but I don't have time to read every post thoroughly, I just cherry pick.
You probably have a lot more people reading your posts who aren't followers. I'd rather have quality than quantity of readers! (Though a high quantity of quality readers would be good too!) But if you want to up the numbers, set up a Facebook page and (on Wordpress at least) your Facebook friends are automatically counted as followers (or so it seems to me?!) Voila - instant followers!
Getting comments is a good indication of readers' engagement - so make lots of opportunities to tempt them to comment, e.g. sign off with open questions at the end of each post.
I'd also recommend constantly tinkering with the look of your blog to keep it fresh and interesting and appealing visually. No matter how good the writing, people won't read it if it's hard on the eye (e.g. I hate white text on black pages). I change my blog quite often. I think a good blog constantly evolves - and as you see more of others' blogs, you'll get more ideas for evolving your own too. Good luck and have fun - blogging IS great fun!