Lightnews — Scholar-powered news

Kevin Schaul @kevinschaul.bsky.social · 1d

That's it for last week. What are you testing?

Kevin Schaul @kevinschaul.bsky.social · 1d

Made a bunch of Sora 2 videos. Mindblowing, terrifying stuff. Here's Sam Altman brewing a fresh pot of AI slop. Lmk if you need an invite code, I have a few left.

7 4

Kevin Schaul @kevinschaul.bsky.social · 1d

Asked claude code to evaluate whether I should migrate my llm evals site to Eleuther’s eval harness. It churned a bit and suggested “inspect ai” instead since I want to include tool usage evals too. Seems like the right call. ✔️

1

Kevin Schaul @kevinschaul.bsky.social · 1d

Tried the same task with Gemini for Chrome. Total failure -- made up a bunch of dates. Seemed like it didn’t have access to click through different links? Not sure what went wrong. ✖️

1

Kevin Schaul @kevinschaul.bsky.social · 1d

Needed to know when something was removed from a website. I asked ChatGPT “agent mode” to visit the url on wayback machine and figure it out. Five minutes later, it gave me the answer. Save me a ton of tedious clicking, and easily verifiable, too. ✔️

1

Kevin Schaul @kevinschaul.bsky.social · 1d

I keep a running log of how AI did on real tasks. This week's notes 🧵

1

Reposted by Kevin Schaul

Block Club Chicago @blockclubchi.bsky.social · 4d

The Block Club Chicago newsroom continues to report on today’s federal immigration activity across Chicago.

If you have footage of agent activity you filmed, please reach out via this website:
blockclubchi.co/3IIYou4

1 53 85

Reposted by Kevin Schaul

Block Club Chicago @blockclubchi.bsky.social · 4d

Federal agents handcuffed Humboldt Park Ald. Jessie Fuentes as she confronted them Friday at Humboldt Park Hospital.

Agents handcuffed her, forced her out of the hospital and threatened her with arrest, Fuentes told a Block Club reporter. blockclubchi.co/3Wkje5V

9 64 98

Reposted by Kevin Schaul

Eileen Clancy 🧿 @clancyny.bsky.social · 4d

Logan Square neighborhood, Chicago. Man wearing a balaclava in a white, unmarked vehicle, pulls the pin on a tear gas canister and tosses it in the road.

"Just trying to grab some lunch and these fucking losers showed up. FUCK ICE!"
source: www.reddit.com/r/LoganSquar...

130 1.3K 2.6K

Reposted by Kevin Schaul

Aaron Reichlin-Melnick @reichlinmelnick.bsky.social · 6d

Surreal moment for America. Needless to say, if the normal police ever pulled something like this — pulling every single person out of an apartment building and handcuffing them to run warrant checks — they would be sued into oblivion.

Yet ICE is going to get away with it entirely.

"It was scary, because I had never had a gun in my face," Fisher said. "They asked my name and my date of birth and asked me, did I have any warrants? And I told them, 'No, 'Ididn't."
Fisher said she was handcuffed before being released around 3 a.m., and she was told that if anyone had any kind of warrant out for them, even if it was unrelated to immigration, they would not be released.

220 2.8K 5.6K

Kevin Schaul @kevinschaul.bsky.social · 6d

www.washingtonpost.com/technology/i...

Inside the secret list of websites that make AI like ChatGPT sound smart

An analysis of a chatbot data set by The Washington Post reveals the proprietary, personal, and often offensive websites that go into an AI’s training data.

www.washingtonpost.com

1 2

Kevin Schaul @kevinschaul.bsky.social · 6d

Here are a couple real answers, if you're still into that whole truth thing wapo.st/3KeqLR0

OpenAI won’t say whose content trained its video tool. We found some clues.

Tests by The Washington Post show OpenAI’s video generator Sora can mimic Netflix shows, TikTok clips and movies, suggesting it used versions of the originals.

wapo.st

1 1 2

Kevin Schaul @kevinschaul.bsky.social · 6d

We're about to get flooded with deepfakes.

Here's an AI-generated clip of me "asking" Sam Altman what they train their systems on (made in 10 seconds with Sora 2)

3 4 14

Kevin Schaul @kevinschaul.bsky.social · 6d

With Sora 2 (launched yesterday), you don't even need the trick. Just type "spongebob explains government shutdown" and it'll happily make it, no questions asked

1

Reposted by Kevin Schaul

Drew Harwell @drewharwell.com · 7d

OpenAI employees are very excited about how well their new AI tool can create fake videos of people doing crimes and have definitely thought through all the implications of this

220 3.3K 11K

Kevin Schaul @kevinschaul.bsky.social · 8d

Just ran some evals on Claude Sonnet 4.5. It's better than 4 on some but worse on a lot. LLM progress is so weird. You really gotta test this stuff on what you care about.

1 4 11

Kevin Schaul @kevinschaul.bsky.social · 8d

Worth a read: OpenAI released an eval for real work tasks across a bunch of industries. They didn't release the individual results (lame), but you can replicate them from the prompts and files.

A usable nugget: If you're outputting pdfs, xlsx or pptx, use Claude.

https://openai.com/index/gdpval/

Chart showing Claude with highest winrate for non-text file extensions

1

Kevin Schaul @kevinschaul.bsky.social · 19d

My favorite nugget is when I tried to make SpongeBob but kept hitting the policy filter.

Thought for a bit, tried "robert the sponge" -- it worked.

AI-generated videos of a SpongeBob-ish character

2 5 25

Kevin Schaul @kevinschaul.bsky.social · 19d

New by me: OpenAI won’t say whose content trained its video tool. We found some clues.

Gift link: wapo.st/3KeqLR0

1 13 34

Kevin Schaul @kevinschaul.bsky.social · 19d

Google's blog post on launching Gemini in Chrome does not include the word "privacy" or "security." Am I missing something or are they not addressing the very real threat of prompt injections?

https://blog.google/products/chrome/new-ai-features-for-chrome/

Screenshot of agentic browsing assistant checking out with Instacart

2

Kevin Schaul @kevinschaul.bsky.social · 21d

Genius LLM eval: Uses Reddit posts on r/AmITheAsshole to test for sycophancy arxiv.org/abs/2505.13995 h/t @gerritd.bsky.social

Social Sycophancy: A Broader Understanding of LLM Sycophancy

A serious risk to the safety and utility of LLMs is sycophancy, i.e., excessive agreement with and flattery of the user. Yet existing work focuses on only one aspect of sycophancy: agreement with user...

arxiv.org

2

Kevin Schaul @kevinschaul.bsky.social · 21d

Meanwhile OpenAI plans to release a different version of ChatGPT for teens openai.com/index/teen-s...

Teen safety, freedom, and privacy

By Sam Altman

openai.com

1

Kevin Schaul @kevinschaul.bsky.social · 21d

New lawsuit against Character AI is “third high-profile case to allege an AI chatbot contributed to a teen’s death by suicide” https://www.washingtonpost.com/technology/2025/09/16/character-ai-suicide-lawsuit-new-juliana/

1 2

Reposted by Kevin Schaul

nitasha tiku @nitasha.bsky.social · 22d

A new wrongful death lawsuit alleges that Character AI and Google are liable for the death by suicide of a 13-year-old girl in Colorado, Juliana Peralta

This is the third wrongful death claim against a popular AI app for a teen's death by suicide this year www.washingtonpost.com/technology/2...

A teen contemplating suicide turned to a chatbot. Is it liable for her death?

A lawsuit filed by the parents of 13-year-old Juliana Peralta against Character AI is the latest to allege a chatbot contributed to a teen’s death by suicide.

www.washingtonpost.com

9 230 510

Reposted by Kevin Schaul

Will Oremus @willoremus.com · Sep 5

Gift link to my developing story on Anthropic's historic settlement with book authors and publishers: wapo.st/47r1gFT

More updates to come.

AI firm Anthropic reaches landmark $1.5B copyright deal with book authors

The company behind the chatbot Claude avoided trial over claims it improperly downloaded millions of books.

www.washingtonpost.com

1 8 18