She doesn't sound controlling at all, possibly she is bitter but she has every right to be.
As someone else said, it is a shame that you did not feel this desperate to see your daughter or work with her best interests before you had an affair.
I am not talking about you being punished but seriously, you have decided that she belongs to you and your new partner, or that is what it sounds like, when in fact she has a mother already. You totally betrayed her mother and now you're trying to take the child away to become part of your 'new' set up, because that suits you, not because it is good for your daughter.
A small child needs stability and one solid grounded stable parent. Two parents is a bonus if they are able to communicate well and they are both stable.
A new baby will be hard for a toddler to deal with in her own environment. Having to deal with a new sibling PLUS a new mother/carer PLUS the only familar person in the house not being THERE most of the time she is visiting PLUS being taken away from her primary carer for a week at a time is likely to cause some serious, serious issues for this poor child.
Think about it, it's not good for her, it's just a fantasy on your part. Why not just accept that you have chosen someone else, have a new child coming along, have moved away and attempted pathetically to justify this with the fact your ex moved away for support she badly needed when you left - she had a good reason, you did not - and that all your choices so far appear to indicate that your older child is in now way your priority any more?
Let her be - yes, skype, and call, and go down when you can. But at least FFS allow her the stability and routine she deserves as a toddler with her mother as her primary carer.
I hope you do not leave the second child as well when it becomes too difficult to cope with things. Nothing to suggest you will but your history does you no favours.
As the first child becomes older things will be much easier to facilitate. You will not lose her entirely if your contact is kept reasonable and realistic at this stage. You will have your chance later but it has to be proportional to what your life choices allow and taking her away for a week (when you won't even be around!!!) is just not proportional AT ALL.