Lightnews — Scholar-powered news

Do LLMs / AI Assistants respect the X-Robots-Tag? - Chris Lever

Chris Green SEO @chris-green.net · 1d

Licensing and gated access for anything you truly cannot risk leaking.
Monitor backlinks pointing at files blocked using x-robots-tag”

If you rely on x-robot tags, and aren't already aware, it's worth a full, deep crawl to examine what these tags being ignored could mean buff.ly/URPn3Ld

🧵 2/2

I asked this recently on LinkedIn and got silence. So let’s dig into it properly here. The question is simple: do LLMs and AI assistants actually respect the X-Robots-Tag? Search engines do. Google,…

buff.ly

Chris Green SEO @chris-green.net · 1d

Chris (another one) questions the reliability of X-robots headers in the world of LLMs/AI search.

"If you’re serious about protecting your data, you need stronger measures:

Explicit robots.txt blocks for AI crawlers that claim to respect them.
Server-level blocks against suspicious IPs.

🧵 1/2

Do LLMs / AI Assistants respect the X-Robots-Tag? - Chris Lever

I asked this recently on LinkedIn and got silence. So let’s dig into it properly here. The question is simple: do LLMs and AI assistants actually respect the X-Robots-Tag? Search engines do. Google,…

buff.ly

GitHub - chonkie-inc/chonkie: 🦛 CHONK docs with Chonkie ✨ — The no-nonsense RAG library

Chris Green SEO @chris-green.net · 2d

This GitHub library is high on my list of libraries to investigate further - github.com/chonkie-inc/...

Github library is here - github.com/chonkie-inc/...

🧵 4/4

🦛 CHONK docs with Chonkie ✨ — The no-nonsense RAG library - chonkie-inc/chonkie

github.com

Chris Green SEO @chris-green.net · 2d

Dan puts it so well (here - www.linkedin.com/pulse/no-non...):

“Tired of making your gazillionth chunker? Sick of the overhead of large libraries? Want to chunk your texts quickly and efficiently? Chonkie the mighty hippo is here to help!”

🧵 3/4

www.linkedin.com

Chris Green SEO @chris-green.net · 2d

But that’s not the only reason you should care about chunking!

If you’re experimenting with tools or analysing content for any reason - especially if you’re sending content to a model for analysis, embeddings, etc., chances are you NEED to worry about chunking.

🧵 2/4

www.linkedin.com

Chris Green SEO @chris-green.net · 2d

You’re fed up with chunking, right? Wait a sec!

There are enough well-respected SEOs who've given sensible accounts for why obsessing over “chunking” your content could be a waste of time from an LLM visibility standpoint. Beyond just readable, “well chunked” content, I'm inclined to agree.

🧵 1/4

www.linkedin.com

Chris Green SEO @chris-green.net · 6d

If you don’t have an AlsoAsked API key (spoilers: it is good), but are willing to help out, send me some keywords which are important, and I’ll give it a test run for you and provide the outputs.

Drop me a comment if you want to help out, and I’ll DM you!

🧵 4/4

Chris Green SEO @chris-green.net · 6d

It’s been a busy time, and I’ve not gone back to it yet, but I’d love some people’s feedback on how they might develop and use “user moments” as part of their content strategies.

🧵 3/4

Chris Green SEO @chris-green.net · 6d

This could also be thought of as something like “keyword clustering”, but rather than just using keywords, it takes PAA questions and clusters them around similar events that may inspire action.

🧵 2/4

Finding User Moments Using PAAs and Also Asked (Work in Progress)

Chris Green SEO @chris-green.net · 6d

Do you have some time to provide some advice? One of the tools I started creating a while back was my People Also Asked (PAA) moments App - www.chris-green.net/post/user-mo...

🧵 1/4

In the Age of AI SEOs NEED to Get Closer to Key User Moments In an age of search which promises a greater degree of personalised results - brought about by AI search - it is critical we understand…

www.chris-green.net

Chris Green SEO @chris-green.net · 6d

Others may not be seen at all, not at least without changes to your current tracking approaches.

If you haven’t had these conversations already, it’s time to start asking who is leading the agent-tracking strategies/initiatives - this one is going to be huge.

🧵 5/5

Chris Green SEO @chris-green.net · 6d

Maybe you aren’t even tracking agents at all
.
The exact picture of tracking agents isn’t fixed. The ones which run from your machine, using your browser, will “look” like you, but they won’t behave like you.

🧵 4/5

Chris Green SEO @chris-green.net · 6d

Agents navigate and engage with your site differently; they could be responsible for significant events that you need to attribute.

But what if you can’t see them? Maybe your conversion rate drops, or your direct revenue increases. Maybe bounce rate changes or pages-per-visit shoot up.

🧵 3/5

Chris Green SEO @chris-green.net · 6d

User-agents like ChatGPT-User are a sign ChatGPT is doing more and more tasks for users, and some of which will include their new agent functions.

These will soon be transacting and booking on behalf of users who may not even visit your site. So you need to be aware of how this works, right?

🧵 2/5

Chris Green SEO @chris-green.net · 6d

“Agents” are all over your site. Well…maybe. But are you even sure how to track them?

If you’re fortunate enough to be able to check your server access logs, you’ll see it happening.

🧵 1/5

AI Search Volatility: Why AI Search Results Keep Changing

Chris Green SEO @chris-green.net · 7d

Citation drift - different sources ranking over time - could be high because their understanding of what users want is developing quickly.

Monitor, observe, try to understand - but focus on satisfying the end-goal of users, not JUST what's rewarded today www.tryprofound.com/blog/ai-sear...

🧵 5/5

New research reveals 'citation drift' in source selection across major AI platforms, showing how AI citations can change by up to 60% in just one month.

AI Search Volatility: Why AI Search Results Keep Changing

Chris Green SEO @chris-green.net · 7d

Does this mean AI search will always be this volatile? Maybe, but you also need to remember this is all very much still in the infant stages. OpenAI specifically is having to learn what does/doesn’t work for users with specific intentions/objectives.

🧵 4/5

New research reveals 'citation drift' in source selection across major AI platforms, showing how AI citations can change by up to 60% in just one month.

AI Search Volatility: Why AI Search Results Keep Changing

Chris Green SEO @chris-green.net · 7d

However, what I have enjoyed about researching AI Overviews and ChatGPT (AND noted is that it is tricky to navigate) is how quickly the data seems to change or how often our own understanding of “AI search” morphs from week to week.

🧵 3/5

New research reveals 'citation drift' in source selection across major AI platforms, showing how AI citations can change by up to 60% in just one month.

AI Search Volatility: Why AI Search Results Keep Changing

Chris Green SEO @chris-green.net · 7d

“Citation drift isn't a bug in AI systems. It's an inherent feature of how these probabilistic models operate. Randomness prevents repetitive responses, pulls in different perspectives, and adapts to changing information landscapes.”

🧵 2/5

New research reveals 'citation drift' in source selection across major AI platforms, showing how AI citations can change by up to 60% in just one month.

AI Search Volatility: Why AI Search Results Keep Changing

1 1 1

Chris Green SEO @chris-green.net · 7d

AI Search is more volatile plays on my mind.

What if you spend months achieving visibility for key events in a user’s journey, only for it to be lost again a few days/weeks later?

www.tryprofound.com/blog/ai-sear... suggests this is more of a feature in AI search, which may be unavoidable.

🧵 1/5

New research reveals 'citation drift' in source selection across major AI platforms, showing how AI citations can change by up to 60% in just one month.

1 1 2

Chris Green SEO @chris-green.net · 8d

Just don’t let a misunderstanding of “cloaking” stop you from considering some simple yet powerful methods to make improvements that your competitors probably aren’t thinking about yet!

🧵 7/7

Chris Green SEO @chris-green.net · 8d

Bot management is moving into a new era of importance, and whilst it can introduce an element of risk, it can also introduce some significant opportunities. SEO has to be interested in bot management, and it should be an active component in shaping approaches and strategies.

🧵 6/7

Chris Green SEO @chris-green.net · 8d

What if that's missing out on the chance to make things better? Google doesn’t need internal tracking parameters on links, and OpenAI’s scrapers don’t need content to be dynamically loaded in. Perhaps you could change what they receive and how they receive it to improve the bot’s experience

🧵 5/7