Your two year old will also likely climb up there. I spent 30 minutes searching for my toddler once, she had climbed up on the top bunk and gone to sleep! But the 2yo could also easily fall off the ladder.
4yos do sleep in bunk beds, midsleepers, high sleepers etc although it isn't recommended. Its a practical solution for a lot of families.