Are you sure that it's not making any difference? I learnt that it's not just about weight some years ago, when I spent 3 months doing some voluntary work, which involved a lot of physical labour. I came back exactly the same weight as when I left - but I was definitely fitter, and my clothes were looser.
I agree that 20 lengths in 30 minutes isn't very fast - if you're used to swimming. If you haven't been doing it for so long, and it's still making you out of breath, then it's fine. Have a target of doing 30 lengths in 30 minutes, but you won't get there overnight - and you say you're already getting faster, so you are progressing. Don't lose sight of that.
How varied is your 20 lengths? Do you just go up and down doing breaststroke at the same pace? If you can do different things, then there's less risk of your body getting into a rut and conditioned to doing the same routine. Try varying it with frontcrawl, and depending on how busy the pool is, backcrawl. Backcrawl is a good stroke, but it's not advisable when there are a lot of other people in.
As well as doing different strokes, you can do exercises to focus on each of them - hold a float in front of you and do legs only, or a float or pullbuoy between your legs and do arms only (not breaststroke, only backcrawl or frontcrawl for that one.)
Also think about varying the speed - sprint for a length. You will probably need to take a breather at the end of the length, if you're working hard - this should leave you very out of breath, however good a swimmer you are; it's your recovery time which will depend on your fitness levels. This is another one you can't really try in a busy pool.
Some of these may depend on how confident a swimmer you are. You could consider a 1 on 1 swimming lesson to get some hints; your pool should be able to advise what's available there. Also, are there aquafit classes available? You don't need to be fit for this, as you can really take it at your own pace, and the water takes a lot of the impact, so it's not too hard on your joints. And it's great fun, splashing about.