Lightnews — Scholar-powered news

Common Crawl Foundation @commoncrawl.bsky.social · 2d

Common Crawl has added IBM’s GneissWeb quality and category annotations to its web dataset, enabling users to filter high-quality content and explore topics like medical, education, and technology.

commoncrawl.org/blog/announc...

Common Crawl - Blog - Announcing GneissWeb Annotations

Common Crawl has added IBM’s GneissWeb quality and category annotations to its web dataset, enabling users to filter high-quality content and explore topics like medical, education, and technology.

commoncrawl.org

1

Common Crawl Foundation @commoncrawl.bsky.social · 6d

Common Crawl’s Web Languages initiative has had many contributions since its introduction. We’re calling for native speakers of certain languages to review language contributions, to ensure that links we’re adding to our seed crawl are of good quality.

commoncrawl.org/blog/web-lan...

Common Crawl - Blog - Web Languages Needing Review by Native Speakers

Common Crawl’s Web Languages initiative has had many contributions since its introduction. We’re calling for native speakers of certain languages to review language contributions, to ensure that links...

commoncrawl.org

4 2

Common Crawl Foundation @commoncrawl.bsky.social · 6d

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August, and September 2025. The host-level graph consists of 628.7 million nodes and 6.9 billion edges, and the domain-level graph consists of 184.6 million nodes and 5.4 billion edges.

Common Crawl - Blog - Host- and Domain-Level Web Graphs July, August, and September 2025

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August, and September 2025. The host-level graph consists of 628.7 million nodes and 6.9...

commoncrawl.org

1 2

Common Crawl Foundation @commoncrawl.bsky.social · 6d

The era of traditional search engine optimization is rapidly evolving into "AIO" (AI optimization), where businesses must ensure their content exists in AI training datasets to remain discoverable as users increasingly turn to AI assistants for answers.

commoncrawl.org/blog/from-se...

Common Crawl - Blog - From SEO to AIO: Why Your Content Needs to Exist in AI Training Data

The era of traditional search engine optimization is rapidly evolving into

commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · 15d

We are pleased to announce the release of our September 2025 crawl, containing 2.39 billion web pages, or 421 TiB of uncompressed content.

www.commoncrawl.org/blog/septemb...

Common Crawl - Blog - September 2025 Crawl Archive Now Available

We are pleased to announce the release of our September 2025 crawl, containing 2.39 billion web pages, or 421 TiB of uncompressed content.

www.commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · 20d

Publishers have been sending Common Crawl legal opt-out requests. In the interest of transparency and to better serve our ecosystem, we are publishing the full opt-out list for every legal request we have received.

commoncrawl.org/blog/common-...

Common Crawl - Blog - Common Crawl Foundation Opt-Out Registry

Publishers have been sending Common Crawl legal opt-out requests. In the interest of transparency and to better serve our ecosystem, we are publishing the full opt-out list for every legal request we ...

commoncrawl.org

1

Common Crawl Foundation @commoncrawl.bsky.social · 20d

On the 28th and 29th of August 2025, Thom Vaughan, Pedro Ortiz Suarez, and Thijs Dalhuijsen attended the Linux Foundation’s AI_dev event in Amsterdam.

commoncrawl.org/blog/trip-re...

Common Crawl - Blog - Trip Report: AI_dev (Linux Foundation) August 2025

On the 28th and 29th of August 2025, Thom Vaughan, Pedro Ortiz Suarez, and Thijs Dalhuijsen attended the Linux Foundation’s AI_dev event in Amsterdam.

commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · 20d

On October 22, the Common Crawl team will lead a seminar at Stanford HAI. Our topic of discussion is “Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data”.

Please register at: hai.stanford.edu/events/commo...

Common Crawl Foundation | Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data | Stanford HAI

Learn about Common Crawl's insights from a recent data product and informed solutions for the future of public web data.

hai.stanford.edu

Common Crawl Foundation @commoncrawl.bsky.social · 29d

We’re Walling Off The Open Internet To Stop AI—And It May End Up Breaking Everything Else

www.techdirt.com/2025/09/08/w...

We’re Walling Off The Open Internet To Stop AI—And It May End Up Breaking Everything Else

A longtime open internet activist recently asked me whether I’d reversed my position on internet openness and copyright because of AI. The question caught me off guard—until I realized what h…

www.techdirt.com

Common Crawl Foundation @commoncrawl.bsky.social · 29d

Stanford HAI and Common Crawl are joining forces to explore how open data can shape the future of AI. On 22 October 2025, their seminar will address privacy, safety, and security while showcasing new ways to preserve and share humanity’s knowledge.
www.commoncrawl.org/blog/common-...

Common Crawl - Blog - Common Crawl Foundation at Stanford HAI: A Shared Legacy of Data and Innovation

Stanford HAI and Common Crawl are joining forces to explore how open data can shape the future of AI. On 22 October 2025, their seminar will address privacy, safety, and security while showcasing new ...

www.commoncrawl.org

1 1

Common Crawl Foundation @commoncrawl.bsky.social · Aug 26

We are pleased to release our newsletter for July and August 2025, with updates on our team's activities.

commoncrawl.org/blog/july-au...

Common Crawl - Blog - July/August 2025 Newsletter

We are pleased to release our newsletter for July and August 2025, with updates on our team's activities.

commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · Aug 22

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July, and August 2025.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - Host- and Domain-Level Web Graphs June, July, and August 2025

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July, and August 2025. The host-level graph consists of 691.1 million nodes and 5.0 bill...

commoncrawl.org

2

Common Crawl Foundation @commoncrawl.bsky.social · Aug 19

We are pleased to announce the release of our August 2025 crawl, containing 2.44 billion web pages (or 424 TiB of uncompressed content).

commoncrawl.org/blog/august-...

Common Crawl - Blog - August 2025 Crawl Archive Now Available

We are pleased to announce the release of our August 2025 crawl, containing 2.44 billion web pages (or 424 TiB of uncompressed content).

commoncrawl.org

1 1

Common Crawl Foundation @commoncrawl.bsky.social · Aug 14

Publishers and brands are shifting from SEO to AIO. Many SEOs unknowingly block their sites from AI search by restricting CCBot in robots.txt. As Search 2.0 transforms discovery, ensuring content can train AI models becomes as crucial as traditional SEO.

commoncrawl.org/blog/ai-opti...

Common Crawl - Blog - AI Optimization Is Here: Are You Ready for Search 2.0?

Publishers and brands are shifting from SEO to AIO. Many SEOs unknowingly block their sites from AI search by restricting CCBot in robots.txt. As Search 2.0 transforms discovery, ensuring content can ...

commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · Aug 14

The Enclosure Of The Open Web And The Open Internet Toll Booth: What’s Behind Pay-By-Crawl

digitalmedusa.org/the-enclosur...

The Enclosure of the Open Web and the Open Internet Toll booth: What’s Behind Pay-By-Crawl - Digital Medusa

Cloudflare recently proposed a system where AI companies and crawlers would pay websites for the right to crawl their content, a move framed as “content independence day”, a response to growing concer...

digitalmedusa.org

Common Crawl Foundation @commoncrawl.bsky.social · Aug 4

A report on IETF 123 in Madrid, including sessions on AI content preferences, bot authentication, and web measurement.

commoncrawl.org/blog/ietf-12...

Common Crawl - Blog - IETF 123 Report

A report on IETF 123 in Madrid, including sessions on AI content preferences, bot authentication, and web measurement.

commoncrawl.org

1 3

Common Crawl Foundation @commoncrawl.bsky.social · Jul 27

Our Web Graph release for July 2025 is now available, consisting of 481.6 million nodes and 3.4 billion edges at the host level, and 209.5 million nodes and 2.6 billion edges at the domain level.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - Host- and Domain-Level Web Graphs May, June, and July 2025

Our Web Graph release for July 2025 is now available, consisting of 481.6 million nodes and 3.4 billion edges at the host level, and 209.5 million nodes and 2.6 billion edges at the domain level.

commoncrawl.org

2

Common Crawl Foundation @commoncrawl.bsky.social · Jul 27

The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content.

commoncrawl.org/blog/july-20...

Common Crawl - Blog - July 2025 Crawl Archive Now Available

The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content.

commoncrawl.org

2

Common Crawl Foundation @commoncrawl.bsky.social · Jul 21

The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identification for web data.

commoncrawl.org/blog/wmdqs-s...

Common Crawl - Blog - WMDQS Shared Task on Language Identification

The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identi...

commoncrawl.org

5 6

Common Crawl Foundation @commoncrawl.bsky.social · Jul 21

"MOIC will also partner with Common Crawl, one of the largest free and open repositories of web crawled data. MOIC will fund work at Common Crawl, leveraging native speakers to annotate and seed European language data in the publicly available Common Crawl data set."

Unlocking data to advance European commerce and culture - Microsoft On the Issues

Microsoft launches 2 initiatives to open Europe’s languages and culture, building on AI, cloud, and digital sovereignty commitments.

blogs.microsoft.com

1 3

Common Crawl Foundation @commoncrawl.bsky.social · Jul 8

In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotations for African languages.

commoncrawl.org/blog/the-fir...

Common Crawl - Blog - The First WMDQS-Masakhane LangID Hackathon

In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotation...

commoncrawl.org

1 2

Common Crawl Foundation @commoncrawl.bsky.social · Jul 2

We are pleased to announce that the Web Graph for June 2025 is now available. The graph consists of 371.6 million nodes and 3.1 billion edges at the host level, and 161.8 million nodes and 2.2 billion edges at the domain level.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - Host- and Domain-Level Web Graphs April, May, and June 2025

We are pleased to announce that the Web Graph for June 2025 is now available. The graph consists of 371.6 million nodes and 3.1 billion edges at the host level, and 161.8 million nodes and 2.2 billion...

commoncrawl.org

Common Crawl Foundation @commoncrawl.bsky.social · Jul 1

The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open source and AI.

commoncrawl.org/blog/common-...

Common Crawl - Blog - Common Crawl at the United Nations Open Source Week, June 2025

The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open s...

commoncrawl.org

2 3

Common Crawl Foundation @commoncrawl.bsky.social · Jun 27

We are pleased to announce that the crawl archive for June 2025 is now available.

www.commoncrawl.org/blog/june-20...

Common Crawl - Blog - June 2025 Crawl Archive Now Available

We are pleased to announce that the crawl archive for June 2025 is now available.

www.commoncrawl.org

1 1

Common Crawl Foundation @commoncrawl.bsky.social · Jun 24

We're happy to share our newsletter for May/June 2025 with updates from our team.

commoncrawl.org/blog/may-jun...

Common Crawl - Blog - May/June 2025 Newsletter

We're happy to share our newsletter for May/June 2025 with updates from our team.

commoncrawl.org

2