It can, but there are limitations ...
How websites can block AI crawlers
- Use robots.txt
A website can publish rules asking recognised AI crawlers not to access particular pages or the entire site. For example, it can name bots used for AI training or AI search.
Limitation: This is a request, not a security barrier. Compliant crawlers respect it; malicious or non-compliant bots can ignore it.
- Block bots at the server or firewall
A website can block known crawler identities, IP addresses, or traffic patterns using a web application firewall (WAF), hosting provider, or CDN.
Limitation: Bots can change IP addresses, disguise their identity, or use infrastructure that makes them difficult to distinguish from legitimate visitors.
- Put content behind a login or paywall
Require authentication, subscriptions, or authorised API access before serving valuable content.
Limitation: This is much stronger than a public crawling directive, provided the server actually enforces access controls. It still cannot prevent an authorised user from copying or sharing material.
- Use bot management and rate limits
Detect automated browsing, throttle repeated requests, challenge suspicious visitors, and limit the volume of data any client can retrieve.
Limitation: Sophisticated crawlers may spread requests across many addresses or imitate human browsing.
To be clear, I'm not advocating any of these - my op can be read literally - I'm simply curious about when the change was introduced and how it was communicated at the time.
I actually think Mnet is s good source of advice, and if it drives traffic to the site then that is a positive result. But I protect myself by changing my username regularly.