Lightnews — Scholar-powered news

David Manheim @davidmanheim.alter.org.il · 3d

Good to see you at least read the abstract.

Now try the paper, especially about how behaviors of attention heads in a model evolve with respect to the training data distribution, and then tell me that language models are directly coded again.

1 1

David Manheim @davidmanheim.alter.org.il · 3d

...what makes someone qualify as a Zionist, in your view?

Because if you mean supporting genocide and/or ethnic cleansing in Gaza, sure, that's obviously horrific. But if you mean wanting a 2-state solution instead of wanting all of Israel wiped off the map, I'm much more concerned.

1

David Manheim @davidmanheim.alter.org.il · 3d

I'm very unsure if you're shockingly ignorant for someone so confident, or shockingly confident for someone so ignorant, but I'm not sure it matters.

Anyways, here's an expert obviously disagreeing with you for you to ignore: arxiv.org/abs/2504.18274

Structural Inference: Interpreting Small Language Models with Susceptibilities

We develop a linear response framework for interpretability that treats a neural network as a Bayesian statistical mechanical system. A small perturbation of the data distribution, for example shiftin...

arxiv.org

1 1

David Manheim @davidmanheim.alter.org.il · 3d

Bottom line: Treat AI-enabled bio risk as rising but still governable, and aim for clearer threat models, empirical monitoring, and adaptive policies.
(12/12)

And if you don't want to read the 100+ page report, read the 6-page @rand.org brief for details:
www.rand.org/pubs/researc...

When Should We Worry About AI Being Used to Design a Pathogen?

Concerns that artificial intelligence (AI) might enable pathogen design are increasing, but risks and timelines remain unclear. This brief describes a Delphi study of biology and AI experts who debate...

www.rand.org

1

David Manheim @davidmanheim.alter.org.il · 3d

And back to the risk, norms matter, but aren’t enough: self-governance (reviews, responsible disclosure) helps, yet can’t reliably constrain determined actors or novel misuse. We’ll need coordinated regulatory and institutional guardrails, though they don't need to be intrusive. (11/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

But even with all the risk, we can invest in pandemic readiness that pays off regardless of origin—rapid diagnostics, scalable vaccines, surge capacity. These reduce incentives and impact even if controls are bypassed.

ASB's recent interview discussed this:
www.youtube.com/watch?v=pnfT...
(10/12)

AI-designed diseases are coming. Here's the defence plan. | Andrew Snyder-Beattie

YouTube video by 80,000 Hours

www.youtube.com

1 1

David Manheim @davidmanheim.alter.org.il · 3d

And to prepare, there are some concrete safeguards to build:
– Strengthen global gene-synthesis screening.
– Add identity checks, experiment pre-screens, and audit trails for cloud/automated labs.
– Improve data governance for genomic/experimental datasets (quality + access control).
(9/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

Policy implications (pragmatic):
– Focus mitigations on plausible, actionable risks and misuse pathways now.
– Risk will increase if barriers fall, so we should monitor four vectors: clinical bioengineering, lab automation, high-fidelity simulations, and generally capable AI.
(8/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

Where expert views diverge: speed of capability gains. Some expect steady, marginal increases; others worry about threshold effects where capabilities jump quickly. Both agree monitoring is essential soon, but there is lots of genuine uncertainty about timelines. (7/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

Biology still pushes back: transmissibility has physical/biological ceilings; environmental stability trades off with other fitness traits, etc. Many constraints interact, so the limits, which we explain at length in the report, need to be understood in concert. (6/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

What AI helps with today: pattern-finding, hypothesis generation, identifying gene targets, and speeding iterative design.

Today, these are force multipliers for sophisticated or state actors, not push-button bioweapons - but no fundamental limits to that happening in the future were found.
(5/12)

1 1

David Manheim @davidmanheim.alter.org.il · 3d

The experts see risk expanding after 2027 as models, automation, and simulation improve.
Until then:
– Data limits: models are only as good as (noisy, incomplete, and biased) data.
– Complex biology: host–pathogen dynamics are hard to predict.
– Wet-lab bottlenecks: validation is slow and expensive.

1 1

David Manheim @davidmanheim.alter.org.il · 3d

I should also clarify: we are explicitly not discussing terrorist misuse of language models to spread extant pathogens. That's unfortunately already possible, especially with jailbroken frontier models - but it is not the biggest risk.

So why the caution (not panic) right now?
(3/12)

1 2

David Manheim @davidmanheim.alter.org.il · 3d

Bottom line (near term): Through ~2027, AI is mostly an accelerator for already-skilled actors—not an autonomous designer of novel pathogens. Useful, yes. Independent, no. And significant uncertainty remains about the slope of progress.
(2/12)

1 2

David Manheim @davidmanheim.alter.org.il · 3d

New RAND report on an important (and messy) question: When should we actually worry about AI being used to design a pathogen? What’s plausible now vs. near-term vs. later?
(1/12)
I helped convene two expert Delphi panels in AI + Bio to weigh in.

Full report:
www.rand.org/pubs/researc...

1 3 5

David Manheim @davidmanheim.alter.org.il · 12d

"Circular financing" and "new revenue helps support NVIDIA’s valuation" is only true if investors think it should be.

So this is actually just saying "NVIDIA investors are happy to fund this deal."

David Manheim @davidmanheim.alter.org.il · 13d

If true, the Dems are so screwed, and that means the US is so screwed.

If they can't go moderate, they will continue to lose elections to populists determined to undermine US democracy generally.

1

David Manheim @davidmanheim.alter.org.il · Sep 7

Link to original: x.com/goodalexande...

Follow-up tweet:

w this framework you can understand:
* Thiel Antichrist obsession
* Kanye getting into Hitler and launching a coin
* Trump memeing himself into becoming President then running again to escape imprisonment
* Elon generating Ani goon slop on the TL
* A16z wilding out

1

David Manheim @davidmanheim.alter.org.il · Sep 7

From the other site - not a complete explanation, but correct as far as it goes:

"the current configuration of economics/ wealth distribution is pretty solidly optimized to drive the wealthiest people in society batshit insane, which - to some extent - explains a lot of things you see around you"

Text:
I think the main reason you find a lot of very unhappy tech people even at the highest levels
- when you're a typical employee everyone around you is making .5-2x what you are
- when you start breaking out wealth goes on log scale. ppl with 10-1000x your net worth become common
- this is native to network effects, scale associated with AI training, and other winner take all dynamics in tech
- all of VC is structured this way as well -- (1 unicorn returns entire fund rest of investments are zero) which psychologically reinforces all or nothing thinking
- this makes competitive people miserable
- this leads them to do hallucinogens or other psychoactive substances in order to accept their place in the universe
- the conclusions drawn from these psychoactive substances are typically at direct odds with how they got to where they are
- and after getting one shotted they're still ultimately in a hard wired competition with people worth 10-1000x more than them
- due to the structure of technology it becomes more or less impossible to break out of your 'bracket' without engaging in increasingly dark things
- you realize that time is running out -- and become aware of synthetic biology (peptides, genetic alteration of children)
- you end up getting involved in police state investments, gooning investments, or crypto -- and view it as non optional to take the gloves off bc everyone around you is doing the same thing
- you're on a permanent hedonic treadmill and you can't ever get off or go back to where you were before bc after doing all of the things you've done you can't possibly ever relate to normal humans
- you get involved with politics or Catholicism or other Lindy cults to try and get off the treadmill
- of course it won't work and you bring all the weird baggage directly into politics or religion and poison those wells too

the current configuration of economics/ wealth distribution is pretty solidly optimized to drive the wealthiest people in...

1 3

David Manheim @davidmanheim.alter.org.il · Sep 4

I've definitely been on videoconferences and listened to talks (while paying attention or event taking notes) while I leave music playing in the background on my headset. And for myself, I don't think it's my verbal IQ being higher - I think it's more the ADHD-motivated stimulus seeking.

4

David Manheim @davidmanheim.alter.org.il · Sep 3

They are good - but they are also mostly rewritten snippets from R:AI-Z, so...

David Manheim @davidmanheim.alter.org.il · Sep 3

...and finally:

David Manheim @davidmanheim.alter.org.il · Sep 3

Looks like this didn't include articles 1-9.

1

Reposted by David Manheim

Margaret Mitchell @mmitchell.bsky.social · Sep 2

🤖📰 Effective YESTERDAY: China has mandated a digital watermark for all AI-generated content.
www.cac.gov.cn/2025-03/14/c...
Translating in 🧵.

Chinese writing, with last phrase highlight.
Something like,

第七条互联网应用程序分发平台在应用程序上架或者上线审核时,应当要求互联网应用程序服务提供者说明是否提供人工智能生务。互联网应用程序服务提供者提供人工智能生成合成服务的,互联网应用程序分发平台应当核验其生成合成内容标识相关材料。
第八条服务提供者应当在用户服务协议中明确说明生成合成内容标识的方法、样式等规范内容,并提示用户仔细阅读并理解相关理要求。
第九条用户申请服务提供者提供没有添加显式标识的生成合成内容的,服务提供者可以在通过用户协议明确用户的标识义务和后,提供不含显式标识的生成合成内容,并依法留存提供对象信息等相关日志不少于六个月。
第十条用户使用网络信息内容传播服务发布生成合成内容的,应当主动声明并使用服务提供者提供的标识功能进行标识。
任何组织和个人不得恶意删除、篡改、伪造、隐匿本办法规定的生成合成内容标识,不得为他人实施上述恶意行为提供工具或者得通过不正当标识手段损害他人合法权益。
第十一条服务提供者开展标识活动的,还应当符合相关法律、行政法规、部门规章和强制性国家标准的要求。
第十二条服务提供者在履行算法备案、安全评估等手续时,应当按照本办法提供生成合成内容标识相关材料,并加强标识信息共范打击相关违法犯罪活动提供支持和帮助。
第十三条违反本办法规定的,由网信、电信、公安和广播电视等有关主管部门依据职责,按照有关法律、行政法规、部门规章的处理。
第十四条本办法自2025年9月1日起施行。

2 55 140

Reposted by David Manheim

koenfucius @koenfucius.bsky.social · Sep 2

When a measure becomes a target, it ceases to be a good measure—Goodhart’s law seems clear enough.

But look closer and you see at least 4 variants, argue Manheim & Garrabrant: Regressional, Extremal, Causal and Adversarial, which offer deeper insight:

buff.ly/h9ORc76

1 3