Definitely. I didn't because that would have meant taking DD camping in March which I think would have been a bit early in the year for me, never mind her. However, plenty of people I know with summer babies have taken their babies even younger and it has been fine.
Basically, their needs are few and basic at that age, and they don't move, so it is pretty easy. You just have to keep them warm at night so plenty of layers are the key. if they are not sleeping with you, then a regular travel cot or a special camping one (maybe pop up) is a good idea. We had the Littlelife Arc 2 and it was fab, giving side access during the night which meant I could give DD a cuddle without having to get out of bed.
Depending how cold it was we used to put DD in a vest, babygro, fleece onesie and either a grobag or blankets. She slept on the cot mattress with an IKEA sheepskin rug on top for extra insulation. She was always toasty. If you can get a onesie with a hood that is great - otherwise a hat might be a good idea as keeping your head warm at night is key with camping.
As far as other campers are concerned, the cries of babies are never as loud as they feel to you in your tent. And even if someone does wake up, they are far more likely just to feel very grateful they are not the ones having to get up, and then turn over and go back to sleep. Really light sleepers on campsites are probably already wearing earplugs against the adult noise and snoring so a baby is unlikely to cause much extra grief.
Definitely do it and have a great time 