Okay, this looks fun :)
If you've got T starfish, C octopuses and L lobsters, then you've got
Eyes = T + 2C + 2L
Legs = 5T + 8C + 10L
But we can simplify the problem by spotting that if we multiply the Eyes equation by 5, we can subtract the Legs equation from it to eliminate both T and L, leaving only C:
5*Eyes = 5T + 10C + 10L
Legs = 5T + 8C + 10L
Subtract one from the other to get:
5*Eyes - Legs = 2C
We used both the Eyes and the Legs equation to derive this one, so we can now throw one of the original ones away. Throw away the Legs equation because it's the messier one - so now we're working with:
5*Eyes - Legs = 2C
Eyes = T + 2C + 2L
We want to find T, C and L in three different scenarios, where all are non-negative integers, and (T + C + L) is prime. Work through them...
Problem 1: The Botany Bay Tank contains 175 legs and 41 eyes, so
2C = 5*Eyes - Legs = 205 - 175 = 30, so C = 15.
T + 2C + 2L = T + 30 + 2L = 41, so T + 2L = 11.
T must be odd (because 2L is even, and 11 is odd), so T could only be 1, 3, 5, 7, 9 or 11. Try them all, checking each time to see if (C+T+L) is prime.
If T = 1 then L = 5. (C+T+L) = 21 which is not prime.
If T = 3 then L = 4. (C+T+L) = 22 which is not prime.
If T = 5 then L=3. (C+T+L) = 23 which is prime.
If T = 7 then L=2. (C+T+L) = 24 which is not prime.
If T = 9 then L=1. (C+T+L) = 25 which is not prime.
If T = 11 then L=0. (C+T+L) = 26 which is not prime.
So the only feasible solution is C=15, T=5, L=3, i.e. 15 octopuses, 5 starfish and 3 lobsters.
Problem 2: The Sargasso Sea Tank contains 92 legs and 22 eyes, so
2C = 5*Eyes - Legs = 110 - 92 = 18, so C = 9.
T + 2C + 2L = Eyes = 22, so T + 2L = 4.
T must be even (because 2L is even and 4 is even), so T could only be 0, 2 or 4. Try them all, checking each time to see if (C+T+L) is prime.
If T = 0 then L = 2. (C+T+L) = 11 which is prime.
If T = 2 then L = 1. (C+T+L) = 12 which is not prime.
If T = 4 then L = 0. (C+T+L) = 13 which is prime.
So there are two feasible solutions:
C=9, T=0, L=2, i.e. 9 octopuses, no starfish and 2 lobsters
C=9, T=4, L=0, i.e. 9 octopuses, 4 starfish and no lobsters.
Problem 3: The La Manche Tank contains 46 legs and 10 eyes, so
2C = 5*Eyes - Legs = 50 - 46 = 4, so C = 2.
T + 2C + 2L = 10, so T + 2L = 6.
T must be even (because 2L is even and 6 is even), so T could only be 0, 2, 4 or 6. Try them all, checking each time to see if (C+T+L) is prime.
If T = 0 then L = 3. (C+T+L) = 5 which is prime.
If T = 2 then L = 2. (C+T+L) = 4 which is not prime.
If T = 4 then L = 1. (C+T+L) = 3 which is prime.
If T = 6 then L = 0. (C+T+L) = 2 which is not prime.
So there are two feasible solutions:
C=2, T=0, L=3, i.e. 2 octopuses, no starfish and 3 lobsters
C=2, T=4, L=1, i.e. 2 octopuses, 4 starfish and 1 lobster.
OP - I don't think I've made any mistakes, but do go through it and check my working. The whole thing took far longer than it should have done because I made a couple of false starts before spotting that simplification at the beginning which made the problem a lot easier to solve. (There might well be quicker ways to do it, but it's the weekend and I've got stuff I need to get on with :) )