Lightnews — Scholar-powered news

Library Innovation Lab @harvardlil.bsky.social · 19h

Our Public Data Project Lead @mollyhardy.bsky.social writes about public data preservation and how it’s complicated by the artificial contemporary distinction between science and the humanities @scholarlykitchen.bsky.social

Guest Post - Rethinking Disciplinary Data Regimes - The Scholarly Kitchen

Between a political policy environment focused on defunding and deleting data collections – an environment in which little can be trusted – and an onslaught of new AI tools that feed indiscriminately ...

scholarlykitchen.sspnet.org

2 1

Library Innovation Lab @harvardlil.bsky.social · Jul 3

Join our team! LIL is looking for a Product and Research Manager to help create, shape, and execute on our portfolio of open knowledge projects. PRMs work across every piece of the LIL ecosystem, from software experimentation to convening of events. Learn more at careers.harvard.edu/job/product-...

Product and Research Manager

careers.harvard.edu

2 4

Library Innovation Lab @harvardlil.bsky.social · Apr 1

Thrilled to share that @maxy.bsky.social's "Century-Scale Storage" was nominated for a Webby Award!

You can vote in the "Best Individual Editorial Feature" category here: vote.webbyawards.com/PublicVoting...

Vote for the best of the internet

I just voted in The Webby People's Voice Awards and checked my voter registration.

vote.webbyawards.com

1 4 12

Reposted by Library Innovation Lab

Greg Leppert @leppert.me · Mar 12

As the @institutionaldatainitiative.org expands its mission, we’re announcing a collaboration with @bpl.boston.gov to develop AI-driven tools capable of accelerating new digitization at libraries across the world, starting at the Boston Public Library. institutionaldatainitiative.org/posts/using-...

Using AI to Accelerate Digitization at Boston Public Librarys

Today, as part of our mission expansion, we’re announcing a collaboration with BPL to develop AI-driven tools capable of accelerating new digitization of large collections at libraries across the worl...

institutionaldatainitiative.org

1 10 18

Reposted by Library Innovation Lab

Greg Leppert @leppert.me · Mar 5

I'm pleased to announce we're expanding our mission at the @institutionaldatainitiative.org with an open call for institutional collaborators, new digitization at Harvard Law School Library, and additional support to advance this work. institutionaldatainitiative.org/posts/open-c...

Expanding Our Mission: An Open Call for Collaborators

Today, we’re pleased to announce an open call for institutional collaborators as new support expands the research capacity of the Institutional Data Initiative.

institutionaldatainitiative.org

1 9 11

Library Innovation Lab @harvardlil.bsky.social · Feb 19

Ed Summers at Stanford wrote this great deep dive of how and why we designed our data.gov archiver the way we did. Thanks for digging in, Ed, this is excellent. inkdroid.org/2025/02/17/n...

Bagging data.gov

inkdroid.org

1 6 18

Library Innovation Lab @harvardlil.bsky.social · Feb 6

We just launched a 16TB archive of every dataset that has been available on data.gov since November. This will be updated day by day as new datasets appear. It can be freely copied, and we're sharing the code behind it to help others make their own archives of data they depend on.

Announcing the Data.gov Archive | Library Innovation Lab

Today we released our archive of data.gov on Source Cooperative. The 16TB collection includes over 311,000 datasets harvested during 2024 and 2025, a complet...

lil.law.harvard.edu

43 1K 1.9K

Reposted by Library Innovation Lab

Lynda M. Kellam #DataRescue @lyndamk.bsky.social · Feb 3

Penn is getting a lot of questions about Data Refuge. That effort no longer exists, but several efforts are currently active. I've created a doc from what I & others have suggested. I'll update as I hear more. Feel free to share or suggest: docs.google.com/document/d/1...

Data Rescue Efforts

Data / Website Rescue Efforts End of Term Crawl - The main coordinated effort to archive websites, but datasets have been more of a challenge. EDGI - They have been focused on environmental data. A ...

docs.google.com

3 45 78

Library Innovation Lab @harvardlil.bsky.social · Jan 31

What are we all missing? Anything you can't get by clicking from link to link like EOT, or downloading datasets directly from data.gov. If there's things you care about preserving that fit that description, that's where to focus.

1 3 11

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Another limited but vital effort is @eotarchive.org. They collect a huge amount from .gov domains every four years, and make it discoverable through @archive.org

End of Term Web Archive

The End of Term Web Archive is a collaborative initiative that collects, preserves, and makes accessible United States Government websites at the end of presidential administrations.

eotarchive.org

6 20

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Our collection from data.gov is limited: if an entry points directly to the data, such as a csv, we have the data. If it points to an html landing page, we just have the landing page. This means many, many datasets are not included. What we have from data.gov adds up to 15 or 20TB.

Data.gov Home - Data.gov

data.gov

1 3 19

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Speaking of telling someone, here’s our update: we have copies of all metadata from data.gov, and all of the dataset URLs it points to (shallow crawl); all federal Github repositories with issues, comments, etc.; and articles from PubMed.

Data.gov Home - Data.gov

data.gov

1 1 17

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Third, tell someone. Archive.org is one good place to store public data for discovery, and we at LIL will consider storing and signing data in some cases as well. Just posting data somewhere search engines can find is good too.

Internet Archive: Digital Library of Free & Borrowable Texts, Movies, Music & Wayback Machine

Archive.org

3 14

Library Innovation Lab @harvardlil.bsky.social · Jan 31

FOIA requests are another great way to scale up — check out @muckrock.com to get started.

2 11

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Next, scale up. If you’re a programmer (or can team up with one), write a python script to download a full collection — say, everything from the data portal of a given government website. Run it yourself, and share it so we libraries can use it too.

3 12

Library Innovation Lab @harvardlil.bsky.social · Jan 31

If you’re a data scientist, good news — your work isn't just downloading data and publishing about it, but also keeping safe copies!

10

Library Innovation Lab @harvardlil.bsky.social · Jan 31

To keep access to stuff you care about: first just make a copy. Use ArchiveWeb.page to click around and download all the parts of a website you’re interested in. We like the desktop version to avoid capturing login cookies or extensions, but the browser extension is good too.

ArchiveWeb.page

5 13

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Public data gets taken down all the time. Everyone needs to understand that there are no complete copies — it belongs to all of us, and we paid for it, but it is too large for anyone to copy. This is why libraries work together to preserve stuff.

1 20

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Why trust us? We’re the web archiving folks behind Perma.cc and the open data folks behind Case.law. Starting late last year we moved into collecting online government datasets, since it can be hard to get the datasets researchers need from web archives.

Websites change. Perma Links don't.

Perma.cc helps scholars, journals, courts, and others create permanent records of the web sources they cite.

Perma.cc

10 32

Library Innovation Lab @harvardlil.bsky.social · Jan 31

Last year we started a project to download and preserve public data. lil.law.harvard.edu/blog/2025/01... Since saving public data is in the news today — but is always needed — let’s talk about what you can do to help.

Preserving Public U.S. Federal Data | Library Innovation Lab

lil.law.harvard.edu

17 200 430

Library Innovation Lab @harvardlil.bsky.social · Jan 22

Could a tool like this help you overcome both language and knowledge barriers when exploring large collections of information? How might LLMs help people access and understand legal information that is either in a foreign language or requires specialized knowledge?

Library Innovation Lab @harvardlil.bsky.social · Jan 22

In this case study, @matteocargnelutti.dev and @kristimukk.bsky.social (in collaboration with Betty Queffelec, University of Western Brittany) investigate how such a tool might help non-French speakers of varying expertise ask questions in English to explore French law.

1

Library Innovation Lab @harvardlil.bsky.social · Jan 22

What insights emerge when a librarian, a software engineer, and a legal scholar come together to experiment with Retrieval Augmented Generation (RAG) to explore over 800,000 French legal articles 🇫🇷?

Blog post: lil.law.harvard.edu/blog/2025/01...
Case study: lil.law.harvard.edu/open-french-...

Title "Open French Law RAG" overlaid on llama dressed in a French lawyer robe in front of an abstract background.

1 3 2

Reposted by Library Innovation Lab

katie mackinnon @ktmac.bsky.social · Dec 19

I’ve been thinking so much lately about data storage, formats, inscription, archives & deep time for the #dataloss project.

So cool to see this piece come out from @harvardlil.bsky.social on CENTURY-SCALE STORAGE by @maxy.bsky.social
(ty @louravn.bsky.social!)

lil.law.harvard.edu/century-scal...

Century-Scale Storage

If you had to store something for 100 years, how would you do it?

lil.law.harvard.edu

1 5 14

Library Innovation Lab @harvardlil.bsky.social · Dec 13

"We picked a century scale because most physical objects can survive 100 years in good care. It is attainable, and yet we selected it because the design of mainstream digital storage mediums are nowhere close to even considering this mark."

lil.law.harvard.edu/century-scale-storage

4 6