Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
October 30, 2025 at 1:21 PM
Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
October 28, 2025 at 1:41 PM
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
Am I Visible on AI : Vérifiez votre visibilité sur ChatGPT#Seo #ChatGPTcrawler #GPTBot #optimisationcontenuIA #robots.txtIA #visibilitéIA
October 24, 2025 at 7:57 AM
Am I Visible on AI : Vérifiez votre visibilité sur ChatGPT#Seo #ChatGPTcrawler #GPTBot #optimisationcontenuIA #robots.txtIA #visibilitéIA
Now that the cat's out of the bag, there should be a way for content creators to prevent their work from being ripped off by billion dollar AI companies.
You can disallow "User-agent: GPTBot" in robots.txt to prevent ChatGPT from ripping off your site, but what about the rest of them?
You can disallow "User-agent: GPTBot" in robots.txt to prevent ChatGPT from ripping off your site, but what about the rest of them?
September 25, 2023 at 11:01 AM
Now that the cat's out of the bag, there should be a way for content creators to prevent their work from being ripped off by billion dollar AI companies.
You can disallow "User-agent: GPTBot" in robots.txt to prevent ChatGPT from ripping off your site, but what about the rest of them?
You can disallow "User-agent: GPTBot" in robots.txt to prevent ChatGPT from ripping off your site, but what about the rest of them?
Their study found that most major AI crawlers can fetch JavaScript files (between 10%-25%) but do not execute it. GPTBot, ClaudeBot, PerplexityBot and more do not currently fully render JavaScript content.
This means that if you're using JS to load content, many AI crawlers will be missing them.
This means that if you're using JS to load content, many AI crawlers will be missing them.
January 7, 2025 at 1:43 PM
Their study found that most major AI crawlers can fetch JavaScript files (between 10%-25%) but do not execute it. GPTBot, ClaudeBot, PerplexityBot and more do not currently fully render JavaScript content.
This means that if you're using JS to load content, many AI crawlers will be missing them.
This means that if you're using JS to load content, many AI crawlers will be missing them.
Schönes Beispiel wie die KI/AI Firmen weiter stehlen OBWOHL da ausdrücklich steht, dass es nicht erlaubt ist:
Quelle: blog.fefe.de?ts=99ac21ad
#künstlicheintelligenz #artificialinteligence #Diebstahl #theft
Quelle: blog.fefe.de?ts=99ac21ad
#künstlicheintelligenz #artificialinteligence #Diebstahl #theft
December 6, 2024 at 10:50 PM
Schönes Beispiel wie die KI/AI Firmen weiter stehlen OBWOHL da ausdrücklich steht, dass es nicht erlaubt ist:
Quelle: blog.fefe.de?ts=99ac21ad
#künstlicheintelligenz #artificialinteligence #Diebstahl #theft
Quelle: blog.fefe.de?ts=99ac21ad
#künstlicheintelligenz #artificialinteligence #Diebstahl #theft
time to nullroute https://openai.com/gptbot-ranges.txt
Similar to approaches taken by search engines, you can now tell OpenAI not to use your site content in its AI models https://platform.openai.com/docs/gptbot
August 11, 2023 at 2:11 PM
time to nullroute https://openai.com/gptbot-ranges.txt
It would be a matter of adding these two lines:
User-agent: GPTBot
Disallow: /
Getting access to edit it would take some effort though: relatively simple through an SEO Wordpress plugin (if you use one), but other than that you’d probably have to download FTP software first, and edit it with that.
User-agent: GPTBot
Disallow: /
Getting access to edit it would take some effort though: relatively simple through an SEO Wordpress plugin (if you use one), but other than that you’d probably have to download FTP software first, and edit it with that.
August 9, 2023 at 8:58 AM
It would be a matter of adding these two lines:
User-agent: GPTBot
Disallow: /
Getting access to edit it would take some effort though: relatively simple through an SEO Wordpress plugin (if you use one), but other than that you’d probably have to download FTP software first, and edit it with that.
User-agent: GPTBot
Disallow: /
Getting access to edit it would take some effort though: relatively simple through an SEO Wordpress plugin (if you use one), but other than that you’d probably have to download FTP software first, and edit it with that.
Writers and editors hate AI for stealing their writings. You can prevent further theft by adding :
User-agent: GPTBot
Disallow: /
In the robots.txt file at the root of your web servers. You can protect yourself if you don't want to train the next models.
User-agent: GPTBot
Disallow: /
In the robots.txt file at the root of your web servers. You can protect yourself if you don't want to train the next models.
September 28, 2023 at 6:37 AM
Writers and editors hate AI for stealing their writings. You can prevent further theft by adding :
User-agent: GPTBot
Disallow: /
In the robots.txt file at the root of your web servers. You can protect yourself if you don't want to train the next models.
User-agent: GPTBot
Disallow: /
In the robots.txt file at the root of your web servers. You can protect yourself if you don't want to train the next models.
« Disruption » : GPTBot, le nouveau robot d’openAI qui aspire les contenus d’internet déjà bloqué par de nombreux sites !
« Disruption » : GPTBot, le nouveau robot d’openAI qui aspire les contenus d’internet déjà bloqué par de nombreux sites !
Malédiction ? Maladie de famille ? Après les nombreuses critiques de non-respect de la propriété intellectuelle ou du RGPD par ChatGPT, à peine né, GPTBot, le robot aspirateur d’OpenAI, suscite déjà la controverse.
www.lesoir.be
September 9, 2023 at 12:08 PM
« Disruption » : GPTBot, le nouveau robot d’openAI qui aspire les contenus d’internet déjà bloqué par de nombreux sites !
it seems that shortly after i wrote this message, GPTBot accesses to my sites have more than doubled thanks, anonymous OpenAI employee??? asie
Interest | Match | Feed
Interest | Match | Feed
Origin
mk.asie.pl
June 27, 2025 at 11:18 AM
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to OEM - GPTBot badness, captions, diversion delay, under-volt, X11 fossil... #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
On Website Technicals (2025-06)
Tech updates: Junited, Cridland, Curran, Bray, Shirriff, DESNZ, GPTBot badness, Siebenmann, Rigby... #Junited2025
www.earth.org.uk
June 29, 2025 at 7:55 PM
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to OEM - GPTBot badness, captions, diversion delay, under-volt, X11 fossil... #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
On Website Technicals (2025-06)
Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #Junited2025
www.earth.org.uk
July 13, 2025 at 7:51 PM
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
CGD has also been slow for the last few days - turns out, among other IPs mining us, OpenAI's GPTBot is hitting us hard - 20,000 times in the last 16 hours. We will be blocking OpenAI's GPTBot forthwith, and hopefully improve the site's responsiveness.
Noticing slowness on the SGD website? Thank the web-scraping AI bots
www.nature.com/articles/d41...
www.nature.com/articles/d41...
Web-scraping AI bots cause disruption for scientific databases and journals
Automated programs gathering training data for artificial-intelligence tools are overwhelming academic websites.
www.nature.com
June 5, 2025 at 5:45 PM
CGD has also been slow for the last few days - turns out, among other IPs mining us, OpenAI's GPTBot is hitting us hard - 20,000 times in the last 16 hours. We will be blocking OpenAI's GPTBot forthwith, and hopefully improve the site's responsiveness.
Pretty extensive robots.txt so far
`DisallowAITraining` is also a newer option, again only for cooperative robots
www.ietf.org/archive/id/d...
`DisallowAITraining` is also a newer option, again only for cooperative robots
www.ietf.org/archive/id/d...
June 11, 2025 at 11:28 AM
Pretty extensive robots.txt so far
`DisallowAITraining` is also a newer option, again only for cooperative robots
www.ietf.org/archive/id/d...
`DisallowAITraining` is also a newer option, again only for cooperative robots
www.ietf.org/archive/id/d...
(위에서 이어짐)
며칠전 블로그 접속 로그 보는데 GPTBot 30개 IP 로 300MB 정도 긁어가더군요. 아이피 조회하니 MS 로 나오더군요. 대역 차단할까하다가 일단 뒀습니다.
robots.txt 로 막을까하다가 예전에 WP 기사 데이터셋 검색에서 제 도메인 이미 나오는것 보고 포기
며칠전 블로그 접속 로그 보는데 GPTBot 30개 IP 로 300MB 정도 긁어가더군요. 아이피 조회하니 MS 로 나오더군요. 대역 차단할까하다가 일단 뒀습니다.
robots.txt 로 막을까하다가 예전에 WP 기사 데이터셋 검색에서 제 도메인 이미 나오는것 보고 포기
July 5, 2024 at 9:11 AM
(위에서 이어짐)
며칠전 블로그 접속 로그 보는데 GPTBot 30개 IP 로 300MB 정도 긁어가더군요. 아이피 조회하니 MS 로 나오더군요. 대역 차단할까하다가 일단 뒀습니다.
robots.txt 로 막을까하다가 예전에 WP 기사 데이터셋 검색에서 제 도메인 이미 나오는것 보고 포기
며칠전 블로그 접속 로그 보는데 GPTBot 30개 IP 로 300MB 정도 긁어가더군요. 아이피 조회하니 MS 로 나오더군요. 대역 차단할까하다가 일단 뒀습니다.
robots.txt 로 막을까하다가 예전에 WP 기사 데이터셋 검색에서 제 도메인 이미 나오는것 보고 포기
You have a different user agent's for search vs GPT.
OAI-Searchbot
Used to link and surface websites in SearchGPT
GPTBot
Used to crawl websites for generative AI foundation models.
Both respect robots.txt
OAI-Searchbot
Used to link and surface websites in SearchGPT
GPTBot
Used to crawl websites for generative AI foundation models.
Both respect robots.txt
November 15, 2024 at 1:49 AM
You have a different user agent's for search vs GPT.
OAI-Searchbot
Used to link and surface websites in SearchGPT
GPTBot
Used to crawl websites for generative AI foundation models.
Both respect robots.txt
OAI-Searchbot
Used to link and surface websites in SearchGPT
GPTBot
Used to crawl websites for generative AI foundation models.
Both respect robots.txt
J'ai découvert ce matin qu'il existait un bot pour chatGPT qui indexait TOUT le contenu des sites pour entraîner son IA. Bref, si vous voulez éviter ça, ajouter cette règles dans votre fichier robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: GPTBot
Disallow: /
September 1, 2023 at 2:16 PM
J'ai découvert ce matin qu'il existait un bot pour chatGPT qui indexait TOUT le contenu des sites pour entraîner son IA. Bref, si vous voulez éviter ça, ajouter cette règles dans votre fichier robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: GPTBot
Disallow: /
I see that openai.com/gptbot is crawling my blog, top to bottom, side to side. I’m sure OpenAI has consulted the “Rights” link clearly displayed on every page, invoking a Creative Commons license that freely grants rights to reuse and remix but not for commercial purposes.
#genAI #llms
#genAI #llms
June 3, 2024 at 5:42 PM
I see that openai.com/gptbot is crawling my blog, top to bottom, side to side. I’m sure OpenAI has consulted the “Rights” link clearly displayed on every page, invoking a Creative Commons license that freely grants rights to reuse and remix but not for commercial purposes.
#genAI #llms
#genAI #llms
How to stop OpenAI from grabbing your site’s content and pouring into ChagGPT. https://searchengineland.com/gptbot-openais-new-web-crawler-430360
GPTBot: OpenAI releases new web crawler
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
searchengineland.com
August 7, 2023 at 7:52 PM
How to stop OpenAI from grabbing your site’s content and pouring into ChagGPT. https://searchengineland.com/gptbot-openais-new-web-crawler-430360
In case it wasn't clear from the robots.txt
@OpenAI
User-agent: GPTBot
Disallow: /
@OpenAI
User-agent: GPTBot
Disallow: /
March 21, 2025 at 2:51 PM
In case it wasn't clear from the robots.txt
@OpenAI
User-agent: GPTBot
Disallow: /
@OpenAI
User-agent: GPTBot
Disallow: /
Has anyone seen the OpenAI web crawler GPTBot visit their site? OpenAI doesn’t follow delay, AFAIK. That was reason I banned theirs schooling bot; it was way too diligent (well, there was anothe ...
Origin | Interest | Match
Origin | Interest | Match
Awakari App
awakari.com
June 23, 2025 at 8:06 PM
Just updated robots.txt.
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗣𝗧𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗼𝗼𝗴𝗹𝗲-𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
LLMs:
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗣𝗧𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗼𝗼𝗴𝗹𝗲-𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
LLMs:
May 1, 2025 at 11:41 AM
Just updated robots.txt.
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗣𝗧𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗼𝗼𝗴𝗹𝗲-𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
LLMs:
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗣𝗧𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗚𝗼𝗼𝗴𝗹𝗲-𝗘𝘅𝘁𝗲𝗻𝗱𝗲𝗱
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
𝗨𝘀𝗲𝗿-𝗮𝗴𝗲𝗻𝘁: 𝗣𝗲𝗿𝗽𝗹𝗲𝘅𝗶𝘁𝘆𝗕𝗼𝘁
𝗗𝗶𝘀𝗮𝗹𝗹𝗼𝘄: /
LLMs:
I'm sorry you feel this way. We really like your site.
Wanted to mention though between several web hosts we tested during the process of approaching how we're now blocking AI crawling, I noted that their AI blocking was actually broken (no robots.txt and User-Agent: GPTBot returns subdomain sites)
Wanted to mention though between several web hosts we tested during the process of approaching how we're now blocking AI crawling, I noted that their AI blocking was actually broken (no robots.txt and User-Agent: GPTBot returns subdomain sites)
January 21, 2025 at 9:25 PM
I'm sorry you feel this way. We really like your site.
Wanted to mention though between several web hosts we tested during the process of approaching how we're now blocking AI crawling, I noted that their AI blocking was actually broken (no robots.txt and User-Agent: GPTBot returns subdomain sites)
Wanted to mention though between several web hosts we tested during the process of approaching how we're now blocking AI crawling, I noted that their AI blocking was actually broken (no robots.txt and User-Agent: GPTBot returns subdomain sites)