But you can have smaller and larger infinities, so what's the issue?
That's a different concept, where we are talking about the sizes of sets and some sets are infinite. There, we can have an infinite set that is bigger than another infinite set, in that however you pair up the two sets, one will have members left over.
Even there, intuition can lead you astray. The set of whole numbers {1, 2, 3, ... } is obviously infinite, as is the set of even numbers {2, 4, 6, ...}. You might think the first set is bigger than the second but it's not because you can pair the sets up:
1 2
2 4
3 6
4 8
But the set of fractions {1/2, 2/3, ... 54/7, ... 962 / 329... } is not bigger than the set of whole numbers.