It might be that trial and error is the easiest, but if you want an algebraic way, you could try this:
x = number of green pens
x + 3 = number of blue pens
2x + 3 = total number of pens.
To find probability of two events, you multiply individual probabilities, so for p(green, green), you get
prob of picking a green pen to start = x/(2x+3)
to find prob of picking a second green pen, you have to subtract 1 from both the number of green pens and number of total pens, so
(x-1)/(2x+2)
Multiply those: x(x-1)/(2x+3)(2x+2)
Do the same for the blue pens:
probability of picking a first blue pen is (x+3)/(2x+3)
subtract 1 from number of blue pens and number of total pens to get the probability of picking a second blue pen: (x+2)/(2x+2)
Multiply those two probabilities to get the probability of picking two blue pens: (x+3)(x+2)/(2x+3)(2x+2).
Then to find the probability of two pens the same colour, you have to add both these possibilities. Luckily you already have a common denominator for the fractions:
[x(x-1) + (x+3)(x+2)]/(2x+3)(2x+2)
Expand the top and bottom
(x2 - x + x2 + 5x +6)/(4x^2 + 10x + 6)
Simplify the top:
(2x2 + 4x +6)/(4x2 + 10x +6)
Further simplify by dividing everything through by 2 (or, factor a 2 out of the top and the bottom, and then cancel, if that is easier to understand)
(x2 + 2x + 3)/(2x2 + 5x + 3)
This is now your expression for picking two pens of the same colour. You can make it into an equation by setting it equal to 27/55
(x2 + 2x + 3)/(2x2 + 5x + 3) = 27/55
Cross multiply (other ways of simplifying are available!)
55 (x2 + 2x + 3) = 27 (2x2 + 5x + 3)
expand
55x2 + 110x + 165 = 54 x2 + 135x + 81
rearrange so that one side = 0
x^2 -25x + 84 = 0
factor
(x-21)(x-4) = 0
x = 21 or x = 4.
Because there are more than 12 pens, x must be 21.
So 21 green, 24 blue, 45 total.
Hope the brackets, division, powers etc makes sense - easier when it's written out as fractions on top of each other.
Other ways of simplifying and solving will also work fine, once you have the basic equation set up.