"Caroline without passport checks, how do you know if the people travelling across a border are Irish, British, or another nationality?"
Exactly as it works now for 167 nations. The UK border is in effect in Dublin, and the Irish border is in effect in Dover. Lets try this.
There is no border between Ireland and NI, yet Vladimir from Russia with his UK visa only is absolutely physically free to wander across to Ireland where, it must be noted, a UK visa is not accepted. Theres not even a signpost telling him hes leaving one for the other.
They do do spot checks, but the reason its not a political issue is what on earth is he going to do? Hes there illegally, but what can our Vlad do? He cant work. He cant get any state support. He cant open a bank account. He cant leave via Ireland. Does he change his name, work illegaly, never needing healthcare and never leave the island again? (As once his UK visa expires and hes not left the UK, hes on the naughty list). His only way out, is the same way he got in, via NI and the UK on a valid UK visa.
Youre conflating the CTA for the wrong purposes. The CTA in one form or another has been in force pretty much since the 1920s. What the current CTA (that you refer to) did, was gain the additional co-operation between UK and Ireland, that ALL travellers would enter one border only when arriving in Ireland/NI/Mainland UK/Jersey/Guernsey/IOM. This was to avoid placing a border control on mainland UK for travellers from the island of Ireland/NI. (Thats been the whole bone of contention, UK Northern Irish, having to show a passport to visit the UK, its absolutely nothing to do with passports on the NI/Ireland border, and its the button the EU want to press by shit stirring suggested 'sea borders')
The easiest and most simplistic way to understand this, is draw a huge line around UK and Ireland, and theres your border, just as it was back in the 1920s when 'no passport control' was first agreed. However, and this is why long posts are absolutely neccesary, Vlad cant fly into Dublin on a UK visa. Ireland doesnt accept them. These sorts of variances, and there are lots of them, are all managed perfectly well.
Coming back to our Vlad though, if someone wants to overstay a tourist trip illegaly they will do. They do it now and theyll do it after Brexit. How do you propose tackling overstayers and how do you think a hard Irish/NI border would have any impact on illegal overstayers whichever side of the border?