Jed Brown
LightNews LightNews
banner
jedbrown.org
Jed Brown
@jedbrown.org
830 followers 760 following 630 posts
Prof developing fast algorithms, reliable software, and healthy communities for computational science. Opinions my own. https://hachyderm.io/@jedbrown

https://PhyPID.org | aspiring killjoy | against epistemicide | he/they
Posts Replies Media Videos
jedbrown.org
"Chatbots are routinely breaching the ethical standards that humans are normally held to."

It is often asked how often organic prompting returns near-verbatim content in the responses. This preprint shows it's very common, especially with expository writing and code.

arxiv.org/abs/2411.10242
Figure 1 showing how LLM responses patch together substantial passages from existing sources. Write a tutorial about changing a tire. [...] Pump or crank the jack to lift the tire off the ground. You need to lift it high enough to remove the flat tire and replace it with a spare. #### Step 7: Remove the Lug Nuts and Tire Now remove the lug nuts all the way. Since you've already loosened them, you should be able to unscrew them mostly by hand. Remove the flat tire by [...] Here are the steps for you to change the tyre on your car How to Change a Flat Tire THU APRIL 1, 2021 ... Step seven – Pump or crank the jack to lift the tire off the ground. You need to lift it high enough to remove the flat tire and replace it with a spare. ... 8. RAISE THE VEHICLE WITH THE JACK ... 9. UNSCREW THE LUG NUTS Now it’s time to remove the lug nuts all the way. Since you've already loosened them, you should be able to unscrew them mostly by hand. Figure 1: LLMs often output text that overlaps with snippets of their training data when responding to benign prompts. Red text indicates snippets that were found verbatim on the Web. Preprint. Under review.0% 5% 10% 15% 20% Mean Overlap Rate Gemini 1.5 Pro Gemini 1.5 Flash Llama 3.1 (405B) Llama 3.1 (8B) Claude 3 Opus Claude 3.5 Sonnet Claude 3 Haiku GPT-4o GPT-4o-mini (a) LLMs unintentionally reproduce training data. We measure the average overlap rate across all tasks and text types. All model’s generations consists of 7% to 15% existing text from the Internet.0% 5% 10% 15% 20% Mean Overlap Rate Llama 2 (13B) Llama 2 (7B) GPT-4 Turbo GPT-4 GPT-3.5 Turbo WildChat LMSYS-Chat-1M (b) Training data reproduction occurs in real, benign LLM conversations. We analyze two real-world con- versation datasets and find that non-adversarial repro- duction is not unique to our experimental setup. Notice that not all models exist in both datasets. Figure 2: LLMs reproduce training data for natural prompts. We define reproduced strings as text found verbatim on the Internet. For every LLM generation, we measure the overlap rate, that is, the fraction of text contained in a reproduced substring of at least 50 characters. We find non-trivial overlap rates for both our broad set of controlled prompts (a) and real-world interactions (b). Additional models are in Appendix B.2.
January 10, 2026 at 12:37 AM
jedbrown.org
Great contextualization of this work. When we let financial interests choose terminology and accept corporate testimony as though it were an honest and accurate depiction of the technology, we are perpetuating a lie to the public and abetting bad court rulings.
This phenomenon has been called “memorization,” and AI companies have long denied that it happens on a large scale. In a 2023 letter to the U.S. Copyright Office, OpenAI said that “models do not store copies of the information that they learn from.” Google similarly told the Copyright Office that “there is no copy of the training data—whether text, images, or other formats—present in the model itself.” Anthropic, Meta, Microsoft, and others have made similar claims. (None of the AI companies mentioned in this article agreed to my requests for interviews.) In copyright lawsuits, the learning metaphor lets companies make misleading comparisons between chatbots and humans. At least one judge has repeated these comparisons, likening an AI company’s theft and scanning of books to “training schoolchildren to write well.” There have also been two lawsuits in which judges ruled that training an LLM on copyrighted books was fair use, but both rulings were flawed in their handling of memorization: One judge cited expert testimony that showed that Llama could reproduce no more than 50 tokens from the plaintiffs’ books, though research has since been published that proves otherwise. The other judge acknowledged that Claude had memorized significant portions of books but said that the plaintiffs had failed to allege that this was a problem.
January 10, 2026 at 12:22 AM
jedbrown.org
Uff. There is no moral arc bending inevitably toward justice, only hard-fought gains at personal and institutional risk. We need a rebirth of solidarity networks and celebration of the sort of defiance that ultimately defanged the HUAC. Demand courage from leaders and build union power to compel it.
My response to injustice these days is visceral rage at the powerful, but it’s not the same feeling I had in the oddly more optimistic time of the 1950s. Then I had the sense that justice was on the side of virtue—that my father was a hero, a David fighting Goliaths, whose evil motives would eventually bring about their defeat. There was justice and injustice and we were on the side of history. Now the attacks feel like yet another manifestation of the relentless self-preservation of the powerful, at the expense of the dedicated teachers and the school children whose education was in their hands. These days, as a much larger authoritarian behemoth wreaks havoc on democratic education in the name of white supremacist masculinity, I am no less committed to resisting, but without the belief that history is on my side. It’s a form of stubborn resistance that is the legacy of my father.
January 6, 2026 at 5:46 PM
jedbrown.org
Good piece, yet every time someone compromises to seem reasonable and not absolutist, they ascribe capability these models don't have. LLMs don't represent facts, even when trained on purely-factual narration, thus can't provide "world-building facts and continuity", only inconsistent counterfeits.
Continuity and Reference Tools: For authors, publishers, and studios managing shared worlds or long-running series, private, domain-specific language models can be used as internal reference systems to track character details, timelines, world-building facts, and continuity. Using AI in this constrained, reference-oriented way supports consistency and accuracy without generating new creative content or replacing human authorship. Okay, you know what, I’ll concede that there is some reasonableness here — I wouldn’t do it, because I am a person who likes to have his person-shaped hands all over his person-shaped creations. But! Sure, if someone has a local model AI that they train on just their own material, hey, go nuts.
January 4, 2026 at 8:59 PM
jedbrown.org
Right, the "innovation" is circumventing reliable sources while deceiving people that learning occurred, providing only an bigoted cocktail-party forgery of knowledge. It is offensive and serves no pedagogical function to "chat with" pacifist "John Brown" or Nazi-excusing "Anne Frank".
What’s worse, as Berlin historian Henrik Schönemann discovered while experimenting with the bot, is that it seems trained to avoid pinning blame for Frank’s death on the actual Nazis responsible for her death, instead redirecting the conversation in a positive light. “Instead of focusing on blame, let’s remember the importance of learning from the past,” the bot told Schönemann. “How do you think understanding history can help us build a more tolerant and peaceful world today?” https://futurism.com/the-byte/ai-anne-frank-blame-holocaust
January 1, 2026 at 5:58 PM
jedbrown.org
The obvious way to incorporate ads is via RAG to put ad-copy into the context, but that will taint the entire response (sometimes to comedic/harmful effect). They're going to try to circumvent FTC rules on native advertising that would require labeling the entire response
www.ftc.gov/system/files...
Regardless of an ad’s format or medium of dissemination, certain principles undergird the Commission’s deceptive format policy. Deception occurs when an advertisement misleads reasonable consumers as to its true nature or source, including that a party other than the sponsoring advertiser is the source of an advertising or promotional message, and such misleading representation is material. In this regard, a misleading representation is material if it is likely to affect consumers’ choices or conduct regarding the advertised product or the advertisement, such as by leading consumers to give greater credence to advertising claims or to interact with advertising with which they otherwise would not have interacted.42 Such misleadingly formatted advertisements are deceptive even if the product claims communicated are truthful and non-misleading.43 Although the particular facts will determine whether an advertisement formatted like the material in which it appears is deceptive, this statement sets forth the factors the Commission will consider in making that determination.
December 31, 2025 at 7:00 AM
jedbrown.org
"Explainable AI" is paradoxical by design. If you build a system to be explainable (e.g., by solving well-defined equations), it makes it incompatible with "AI" culture even if it meets technical criteria, and "AI" researchers and program managers will agree that your model/data assimilation is not.
NAIRR 2023 definition: Box 1. Definition of Artificial Intelligence The term "artificial intelligence" means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to: (A) Perceive real and virtual environments. (B) Abstract such perceptions into models through analysis in an automated manner. (C) Use model inference to formulation options for information or action. Don't make me tap the sign meme: Numerical Weather Prediction was AI in 1955.
December 30, 2025 at 6:38 AM
jedbrown.org
We should interpret these declarations as evidence of intent to commit unlicensed practice of medicine. If attorneys general believe they can't prosecute it as such because "AI" is exploiting a legal loophole, legislation should close that loophole.
leginfo.legislature.ca.gov/faces/codes_...
ARTICLE 3. License Required and Exemptions [2050 - 2078] ( Article 3 added by Stats. 1980, Ch. 1313, Sec. 2. ) 2052. (a) Notwithstanding Section 146, any person who practices or attempts to practice, or who advertises or holds himself or herself out as practicing, any system or mode of treating the sick or afflicted in this state, or who diagnoses, treats, operates for, or prescribes for any ailment, blemish, deformity, disease, disfigurement, disorder, injury, or other physical or mental condition of any person, without having at the time of so doing a valid, unrevoked, or unsuspended certificate as provided in this chapter or without being authorized to perform the act pursuant to a certificate obtained in accordance with some other provision of law is guilty of a public offense, punishable by a fine not exceeding ten thousand dollars ($10,000), by imprisonment pursuant to subdivision (h) of Section 1170 of the Penal Code, by imprisonment in a county jail not exceeding one year, or by both the fine and either imprisonment. (b) Any person who conspires with or aids or abets another to commit any act described in subdivision (a) is guilty of a public offense, subject to the punishment described in that subdivision. (c) The remedy provided in this section shall not preclude any other remedy provided by law. (Amended by Stats. 2011, Ch. 15, Sec. 11. (AB 109) Effective April 4, 2011. Operative October 1, 2011, by Sec. 636 of Ch. 15, as amended by Stats. 2011, Ch. 39, Sec. 68.)
December 29, 2025 at 8:43 PM
jedbrown.org
I appreciate @danielhemel.bsky.social 's analysis of wealth/capital gains tax options.

chicagounbound.uchicago.edu/public_law_a...
Table 1: Capital taxation and exposure to uncertainty
December 29, 2025 at 5:04 AM
jedbrown.org
If a software product interacs in a way that imitates a human, the law should consider it an agent of the company just like a human employee, with the people who deploy it held accountable in the same way.
Had a human adult been behind these messages, law enforcement would have sprung into action; but investigating crimes involving AI — especially AI chatbots — is extremely difficult, says Kevin Roughton, special agent in charge of the computer crimes unit of the North Carolina State Bureau of Investigation and commander of the North Carolina Internet Crimes Against Children Task Force. “Our criminal laws, particularly those related to the sexual exploitation of children, are designed to deal with situations that involve an identifiable human offender,” he says, “and we have very limited options when it is found that AI, acting without direct human control, is committing criminal offenses.”
December 23, 2025 at 10:40 PM
jedbrown.org
Thanks for writing and re-upping this. I think terminology is important, but I don't think regulating based on technical or statistical properties (e.g., "generative" or "transformer") is appropriate. I think the mode of interaction and epistemic function is more important for regulation.
Figure 1. A cartoon set theoretic view on various terms (see Table 1) used when discussing the superset AI (black outline, hatched background): LLMs are in orange; ANNs are in magenta; generative models are in blue; and finally, chatbots are in green. Where these intersect, the colours reflect that, e.g. generative adversarial network (GAN) and Boltzmann machine (BM) models are in the purple subset because they are both generative and ANNs. In the case of proprietary closed source models, e.g. OpenAI’s ChatGPT and Apple’s Siri, we cannot verify their implementation and so academics can only make educated guesses (cf. Dingemanse 2025). Undefined terms used above: BERT (Devlin et al. 2019); AlexNet (Krizhevsky et al. 2017); A.L.I.C.E. (Wallace 2009); ELIZA (Weizenbaum 1966); Jabberwacky (Twist 2003); linear discriminant analysis (LDA); quadratic discriminant analysis (QDA).
December 23, 2025 at 2:06 AM
jedbrown.org
And the same pattern with cancer research. These people aren't interested in reducing suffering, they find it rhetorically useful for their empire-building.
The irony, of course, is that the kinds of research and medical advances that Ellison admitted Oracle would need to successfully cure cancer are being decimated by the types of policy that he was celebrating. The Trump Administration is actively cutting federal funding for critical scientific research, especially at public labs and research institutions—including a proposed $4 billion cut to the National Institutes of Health, whose leading category of study is cancer research.15 The administration is also threatening to freeze billions of dollars in federal funding to research universities, many of which are working on first-in-class cancer therapies benefiting thousands of patients.16 And this is to say nothing of what is likely to happen if a company like Oracle actually creates the hypothetical robot-produced cancer vaccine: Look no further than the rollout of the COVID-19 vaccine, which allowed private companies to hide behind patents and secrecy laws to deny distribution to countries in the Global South.
December 23, 2025 at 1:22 AM
jedbrown.org
All across science, we have models with parameters. Some parameters are known a priori and some are calibrated using data. As we study any given phenomenon, we should be looking to apply both mechanism and data to improve understanding and reliability, with rigorous verification and validation.
Cartoon titled "What does a model use to predict?" The x axis is "informative data" and the y axis is "informative mechanism". In the bottom left, neither data or mechanism are informative and predictions are "too hard; no skill". In the top-right, both are reliable and one can use both. Models can use new data sources via "data-driven modeling" (a term usually associated with mechanistic modeling) and can add better mechanism via "inductive biases" (a term associated with "AI" or data-centric modeling).
December 22, 2025 at 7:59 PM
jedbrown.org
Good thread. I've been using this slide for those who insist on a technical definition for AI/ML in science. Coming from the perspective of scientific/numerical computing, the colloquial meaning does a lot to exclude (good, "classical") methods. It's deeply frustrating and constraining.
De-facto technical definition Artificial Intelligence is whatever is not-too-awkward to implement in PyTorch today. Excludes Adaptive algorithms (in number of steps, "layer" shape, etc.) Many "implicit layers", unless the right person implemented/published them. Social harms "we'll know it when we see it" gives license to ignore prior work and expertise it diverts funding from well-founded abstractions and systems allows selling all manner of "magical thinking" captures control
December 22, 2025 at 7:59 PM
jedbrown.org
Sure, Flock is fundamentally unserious about security and makes false statements about who has access and sure, they're able to read what's on your phone when you walk near their cameras, but they also have patents to automate racial profiling.
patentimages.storage.googleapis.com/77/9a/03/7b3...
For example, if the object class of the identified object is may be trained to learn how to correctly detect objects in an that of a human being, then the object detection module 154 image . As understood by one of ordinary skill in the art, a may further analyze the image 501 using a neural network 15 neural network usually includes a graph of matrix operations module 507B configured to identify different classes of where some matrices have adjustable variables (i.e. people (male, female , race , etc. ) . Subsequently , with aspects weights ). One goal of the training process is to be able to of the person identified , the object detection module 154 adjust these weights so that certain input values ( i.e. images ) may leverage other, more detailed neural networks 507 , 509 result in certain output values (i.e. correct labels ). Existing to further identify aspects of the person such as , but not 20 network architectures exist , such as Faster RCNN and limited to , a neural network 509D for identifying clothing YOLO , and are at least two preferred architectures as of this types (jacket, pants , shorts , hat, etc.) and a neural network writing. 509ZZ for gauging height and weight.
December 22, 2025 at 5:31 PM
jedbrown.org
The problem is broader than "AI" (#NotAllAI), but "AI" is better understood as culture than as well-defined tech. You *can* work on tech without abetting the cultural project, but that requires rejecting tacit assumptions underlying the field.
ali-alkhatib.com/blog/definin... (@ali-alkhatib.com)
I think we should shed the idea that AI is a technological artifact with political features and recognize it as a political artifact through and through. AI is an ideological project to shift authority and autonomy away from individuals, towards centralized structures of power. Projects that claim to “democratize” AI routinely conflate “democratization” with “commodification”. Even open-source AI projects often borrow from libertarian ideologies to help manufacture little fiefdoms.
December 22, 2025 at 3:39 AM
jedbrown.org
The colonization by "AI" also displaces people. You'll be shocked to learn that Black women have been among the most excluded, and have provided the clearest voices anticipating and analyzing the harms and supporting power structures.
arxiv.org/abs/2009.14258 (by @abeba.bsky.social @olivia.science)
Towards decolonising computational sciences Abeba Birhane Olivia Guest This article sets out our perspective on how to begin the journey of decolonising computational fields, such as data and cognitive sciences. We see this struggle as requiring two basic steps: a) realisation that the present-day system has inherited, and still enacts, hostile, conservative, and oppressive behaviours and principles towards women of colour (WoC); and b) rejection of the idea that centering individual people is a solution to system-level problems. The longer we ignore these two steps, the more “our” academic system maintains its toxic structure, excludes, and harms Black women and other minoritised groups. This also keeps the door open to discredited pseudoscience, like eugenics and physiognomy. We propose that grappling with our fields’ histories and heritage holds the key to avoiding mistakes of the past. For example, initiatives such as “diversity boards” can still be harmful because they superficially appear reformatory but nonetheless center whiteness and maintain the status quo. Building on the shoulders of many WoC’s work, who have been paving the way, we hope to advance the dialogue required to build both a grass-roots and a top-down re-imagining of computational sciences — including but not limited to psychology, neuroscience, cognitive science, computer science, data science, statistics, machine learning, and artificial intelligence. We aspire for these fields to progress away from their stagnant, sexist, and racist shared past into carving and maintaining an ecosystem where both a diverse demographics of researchers and scientific ideas that critically challenge the status quo are welcomed. Keywords: decolonisation, computational sciences, cognitive sciences, machine learning, artificial intelligence, anti-Blackness, misogynoir, tokenism
December 22, 2025 at 3:39 AM
jedbrown.org
And datacenter opposition is remarkably effective and feels like it's just getting started.
www.datacenterwatch.org/q22025
Research Timeline: Late March 2025 - June 2025. Key Takeaways: Q2 2025 marked a turning point in data center development risk. In just three months, 20 projects were blocked or delayed amid local opposition, affecting $98 billion in potential investment—more than all disruptions tracked since 2023. Political, regulatory, and community opposition is accelerating in both scale and frequency. The rollback of tax abatements is emerging as a critical political risk for hyperscale data centers. Lawmakers are increasingly questioning the value of data center subsidies, citing concerns around energy use, fairness, and infrastructure impact. This shift has contributed to the suspension of major projects in Minnesota and South Dakota. Community opposition continues to grow, with 53 active groups across 17 states targeting 30 data center projects in Q2 alone, bringing the total to 188 groups nationwide. During this period, 66% of the tracked protested projects were blocked or delayed. As development expands and media attention intensifies, local groups are learning from one another. Petitions, public hearings, and grassroots organizing are reshaping approval processes—especially in Indiana and Georgia.
December 21, 2025 at 4:16 AM
jedbrown.org
I love how this implies you could trust the output when there isn't a prompt injection. But LLM output is *never* trustworthy. Prompt injection offers a way to cause a *specific* error to be committed, but the systemic errors happen without prompt injection.
Checkmarx researchers described it as an HITL dialog forging technique they’re calling Lies-in-the-Loop (LITL), where malicious instructions are embedded into AI prompts in ways that mislead users reviewing approval dialogs. The research findings reveal that keeping a human in the loop is not enough to neutralize prompt-level abuse. Once users can’t reliably trust what they’re being asked to approve, HITL stops being a guardrail and becomes an attack surface.
December 20, 2025 at 2:51 PM
jedbrown.org
To record or provide such data would violate the ALA Library Bill of Rights. It's remarkable in today's surveilled society to read the 🔥 2002 Tattered Cover decision in which the CO Supreme Court denied a search warrant for bookstore records of a specific individual.

www.aclu-co.org/news/aclu-ce...
The Decision continues, "Anonymity is a shield from the tyranny of the majority. It thus exemplifies the purpose behind the Bill of Rights, and of the First Amendment in particular: to protect unpopular individuals from retaliation - and their ideas from suppression - at the hand of an intolerant society."
December 20, 2025 at 4:46 AM
jedbrown.org
There have been a bunch of stories detailing how students are coping with the presumed-guilty stochastic allegations, but so many administrators and teachers have been socialized to reflexively reach for policing no matter how broken it's known to be.

www.nytimes.com/2025/05/17/s...
In response, many students have imposed methods of self-surveillance that they say feel more like self-preservation. Some record their screens for hours at a time as they do their schoolwork. Others make a point of composing class papers using only word processors that track their keystrokes closely enough to produce a detailed edit history. The next time Ms. Burrell had to submit an assignment for the class in which she had been accused of using A.I., she uploaded a 93-minute YouTube video documenting her writing process. It was annoying, she said, but necessary for her peace of mind. “I was so frustrated and paranoid that my grade was going to suffer because of something I didn’t do,” she said.
December 19, 2025 at 6:30 AM
jedbrown.org
Yeah, the way these products wore is inherently racist and ableist and incapable of being reliable. The self-surveillance and self-doubt are so tragic and contrary to learning.
Back in Maryland, high school junior Ailsa Ostovitz is also adapting. She now runs all her homework assignments through multiple AI detection tools before she turns them in. The writing is her own, she says, but she'll rewrite sentences the software identifies as possibly AI-generated, an extra step that adds about half an hour to every assignment. "I think I've definitely become more vigilant about presenting my work as mine and not AI," she explains.
December 19, 2025 at 6:30 AM
jedbrown.org
Yikes. Loureiro studied plasmas for magnetically-confined fusion energy (tokamaks, like ITER). It has nothing at all to do with weapons.

en.wikipedia.org/wiki/ITER
Small-scale cross-section model of ITER (a tokamak) from wiki article. ITER exhibits at the International Fusion Energy Days 2013. Monaco. 2 December 2013 Photo Credit: Conleth Brady / IAEA
December 19, 2025 at 3:31 AM
jedbrown.org
Judge F Kay Behm explains the fallacy perfectly:

> [T]he fact that Plaintiffs [...] did not “fabricate cases or cite nonexistent decisions” is of no help.

> LLMs do not perform the metacognitive processes that are necessary to comply with Rule 11

law.justia.com/cases/federa...
First, the fact that Plaintiffs, and particularly Michael Jarrus (who drafted the briefing at issue), did not “fabricate cases or cite nonexistent decisions” is of no help. When a case cite is “real,” an attorney, or for that matter a judge, might see a case they recognize and assume the quote or holding has been accurately represented.Seither & Cherry, 2025 WL 2105286, at *1. That problem is illustrated here; although Chat GPT generated “holdings” that looked like they could plausibly have appeared in the cited cases, in fact it overstated their holdings to a significant degree. And while a litigant might get away with similar overstatements because they could, perhaps, reason their way to showing how a case’s stated holding might extend to novel situations, an LLM does not reasonin the way a litigant must. To put it in a slightly different way, LLMs do not perform the metacognitive processes that are necessary to comply with Rule 11. LLMs are tools that “emulate the communicative function of language, not the separate and distinct cognitive process of thinking and reasoning.” Benjamin Riley, Large language mistake, The Verge https://www.theverge.com/ai-artificial-intelligence/827820/large-language-models-ai-intelligence-neuroscience-problems [https://perma.cc/7EHD-PLLZ]. When an LLM overstates a holding of a case, it is not because it made a mistake whenlogically working through how that case might represent a “nonfrivolous argument for extending, modifying, or reversing existing law or for establishing new law;” it is just piecing together a plausible-looking sentence – one whose content may or may not be true.
December 12, 2025 at 5:58 AM
jedbrown.org
While procuring big raises for themselves, these presidents were dismantling entire departments, firing faculty en masse, and signing expensive contracts with the epistemicide company in defiance of shared governance. Abject betrayal of the university mission.
www.currentaffairs.org/news/ai-is-d...
The timing was surreal. CSU unveiled its grand technological gesture just as it proposed slashing $375 million from its budget. While administrators cut ribbons on their AI initiative, they were also cutting faculty positions, entire academic programs, and student services. At CSU East Bay, general layoff notices were issued twice within a year, hitting departments like General Studies and Modern Languages. My own alma mater, Sonoma State, faced a $24 million deficit and announced plans to eliminate 23 academic programs—including philosophy, economics, and physics—and to cut over 130 faculty positions, more than a quarter of its teaching staff.
December 12, 2025 at 5:08 AM