Lightnews — Scholar-powered news

David Huang @davidhuang.blog · 3d

Sample inefficiency??

Tim Kellogg @timkellogg.me · 3d

Dario Amodei:

“100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it.

And AI Models actually do learn during the context window, without changing the weights”

x.com/rohanpaul_ai...

Rohan Paul on X: ""100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it. And AI Models actually do learn during the context window, without changing the weights." ~ Anthropic CEO Dario https://t.co/vGEW7iNGLD" / X

"100 million words context window is already possible, which is roughly what a human hears in a lifetime. Inference support is the only bottleneck to achieve it. And AI Models actually do learn during the context window, without changing the weights." ~ Anthropic CEO Dario https://t.co/vGEW7iNGLD

x.com

1

David Huang @davidhuang.blog · 4d

Suppose your model of intelligence requires (1) NOT imitation learning (2) NOT virtual environments, and (3) NOT compressed timelines, then what BENEFIT is there to training an “artificial” intelligence vs an “organic” intelligence.

Isn’t the organic more good?

1 1

David Huang @davidhuang.blog · 26d

I think you learn a lot about a person when you learn what is the tiny garden they will protect from slop and where they will let slop go wild.

Reposted by David Huang

Tim Kellogg @timkellogg.me · 29d

a good breakdown of the Claude Code agent loop

the key: a simple 2-layer architecture

“a … single-threaded master loop (codenamed nO) real-time steering (the h2A queue)”

(yes, they totally read a de-compiled/obfuscated version, but yeah dedication!)

blog.promptlayer.com/claude-code-...

Claude Code: Behind-the-scenes of the master agent loop

When Tom's Guide reported that Anthropic had to add weekly limits after users ran Claude Code 24/7, it caused quite a stir. Claude Code did something right. Let's dive into the architecture behind the...

blog.promptlayer.com

3 7 35

David Huang @davidhuang.blog · Sep 5

Too much spam to take notes this way… maybe more subdomains is the answer

David Huang @davidhuang.blog · Sep 1

Russell, Bertrand. The History of Western Philosophy

David Huang @davidhuang.blog · Sep 5

He does know he is trying for the Nobel PEACE Prize right? Like put some effort into it. Just as easily could have been the Department of PEACE.

1

David Huang @davidhuang.blog · Sep 4

Ayy, thanks

1

David Huang @davidhuang.blog · Sep 4

Ok yeah but if I’m understanding right (maybe not relevant to GM, more just about general trends) it’s not about “scaling”, but rather it’s about data mix/ quality, algos and architectures?

1 3

David Huang @davidhuang.blog · Sep 4

I’m dense … what is he wrong about?

My read is that if GPT-5 is scaled up then pure scaling does not work (from evaluating the model… and if it is not scaled up then OAI must have try pure scaling up and it did not work.

1 1

David Huang @davidhuang.blog · Sep 1

This is an example of the evils of specialization: a man must not write on Plato h less he has spent so much of his youth on Greek as to have no time for the things that Plato thought important.”

David Huang @davidhuang.blog · Sep 1

P.132 “It is noteworthy that modern Platonists, without exception are ignorant of mathematics, in spite of the immense importance that Plato attached …

1

David Huang @davidhuang.blog · Sep 1

more indefensible? Is there any position that is, in practice, more required for the purposes of sane and productive development?

David Huang @davidhuang.blog · Sep 1

p. 117-8 "Plato is convinced that there is 'the Good,' and that its nature can be ascertained, when people disagree about it, one, at least, is making an intellectual error, just as much as if the disagreement were a scientific one on some matter of fact"

Is there any position that is, in practice,

1

David Huang @davidhuang.blog · Sep 1

LLMs and the implantation of certain "myths" by certain folks...

Conceptually lies in opposition to the idea of certain (successful) groups having shibboleths and common lies.

David Huang @davidhuang.blog · Sep 1

p. 113 "Plato is right in thinking that belief in this myth could be generated in two generations ... What Plato does not seem to realize is that the compulsory acceptance of myths is incompatible with philosophy and involves a kind of education that stunts intelligence."

Clear "application" to...

1

David Huang @davidhuang.blog · Sep 1

calculating?

Can calculating be construed in the broadest sense? The evolutionary calculus picking out the most propagating or preserving stories? How does this effect our analysis of the stories that remain? How does that counter moral considerations?

David Huang @davidhuang.blog · Sep 1

taking, etc.

Most things seem profoundly calibrated against death.

Massive healthcare spend. Insane efforts. (insert cartoon of Trump w Covid on balcony). Million dollar doses.

See also, what are things that are calculated in our media today? What are they calculated to do? Who is doing the...

1

David Huang @davidhuang.blog · Sep 1

p. 109 "There are things in Homer and Hesiod, which are calculated to make their readers fear death, whereas everything ought to be done in education to make young people willing to die in battle."

Notes: see in opposition to "Don't Die" Bryan Johnson, Singer to the extreme, welfare states, hostage

1

David Huang @davidhuang.blog · Sep 1

Elon Musk, etc.

I think in the 2025 context you especially see the decoupling of birthrate and military power. Seems more capital and technology bound, i.e. sword missiles falling from outer space to hit internet-located targets... drones, ai surveillance, etc.

1 1

David Huang @davidhuang.blog · Sep 1

p. 102 "Lycurgus, like other reformers, thought the education of children "the chiefest and greatest matter, that a reformer of laws should establish'; and like all who aim chiefly at military power, he was anxious to keep up the birthrate."

Interesting re. AI, birthrate discourse, modern wars...

1 1

David Huang @davidhuang.blog · Sep 1

Russell, Bertrand. The History of Western Philosophy

5

David Huang @davidhuang.blog · Aug 29

I use ChatGPT with memory off and no longer than 3 back and forth messages

Philip Stafford @staffordphilip.bsky.social · Aug 29

Awful story, again ChatGPT at the centre of it. The cost of a 'growth at any cost' SV culture.

A Troubled Man, His Chatbot and a Murder-Suicide in Old Greenwich
www.wsj.com/tech/ai/chat...

1

Reposted by David Huang

Philip Stafford @staffordphilip.bsky.social · Aug 29

Awful story, again ChatGPT at the centre of it. The cost of a 'growth at any cost' SV culture.

A Troubled Man, His Chatbot and a Murder-Suicide in Old Greenwich
www.wsj.com/tech/ai/chat...

2 12

David Huang @davidhuang.blog · Aug 20

I see this as a sort of "LLM as a judge" in the judicial family court and arbitration sense (arbitration broadly speaking)

David Huang @davidhuang.blog · Aug 8

The problem with “reasoning” models is a limited conception of reasoning.

Math/CS focus make it easy for it to understand internally consistent realms, and sensemake there.

It also make it easier to have psychotic breaks.

But harder to make sense of cloudy information.

1