Curtis Weyant
@curtisweyant.com
Writer, Editor & Content Strategist | SEO Expert & Trainer | Nerdy Book Stuff @curtdubya.bsky.social
I tried blocking AI bots from crawling my website using the list at github.com/ai-robots-txt/
The list includes several Google UAs:
Google-CloudVertexBot
Google-Extended
GoogleOther[-Image|-Video]
None of these should affect Google Search according to developers.google.com/search/docs/...
The list includes several Google UAs:
Google-CloudVertexBot
Google-Extended
GoogleOther[-Image|-Video]
None of these should affect Google Search according to developers.google.com/search/docs/...
ai.robots.txt
A community effort to identify and block AI crawlers. - ai.robots.txt
github.com
July 21, 2025 at 12:15 PM
I tried blocking AI bots from crawling my website using the list at github.com/ai-robots-txt/
The list includes several Google UAs:
Google-CloudVertexBot
Google-Extended
GoogleOther[-Image|-Video]
None of these should affect Google Search according to developers.google.com/search/docs/...
The list includes several Google UAs:
Google-CloudVertexBot
Google-Extended
GoogleOther[-Image|-Video]
None of these should affect Google Search according to developers.google.com/search/docs/...
Reposted by Curtis Weyant
The work is indeed mysterious and important
June 15, 2025 at 6:24 AM
The work is indeed mysterious and important