Lightnews — Scholar-powered news

LightNews

Javier Rando

@javirandor.com

260 followers 97 following 45 posts

Red-Teaming LLMs / PhD student at ETH Zurich / Prev. research intern at Meta / People call me Javi / Vegan 🌱 Website: javirando.com

Posts Media Videos Starter Packs

Pinned

Javier Rando @javirandor.com · Nov 25

Anyone may be able to compromise LLMs with malicious content posted online. With just a small amount of data, adversaries can backdoor chatbots to become unusable for RAG, or bias their outputs towards specific beliefs. Check our latest work! 👇🧵

Javier Rando @javirandor.com · Feb 18

Thank you so much for the invite!

Javier Rando @javirandor.com · Feb 10

We really hope this analysis can help the community better understand where we come from, where we stand, and what things may help us make meaningful progress in the future.

Co-authored with @jiezhang-ethz.bsky.social, Nicholas Carlini and @floriantramer.bsky.social

arxiv.org/abs/2502.02260

Adversarial ML Problems Are Getting Harder to Solve and to Evaluate

In the past decade, considerable research effort has been devoted to securing machine learning (ML) models that operate in adversarial settings. Yet, progress has been slow even for simple "toy" probl...

Javier Rando @javirandor.com · Feb 10

We propose that adversarial ML research should clearly differentiate between two problems:

1️⃣ Real-world vulnerabilities. Attacks and defenses on ill-defined problems are valuable when harm is immediate.

2️⃣ Scientific understanding. We should study well-defined problems.

Javier Rando @javirandor.com · Feb 10

We are aware that this is not a simple problem and some changes may actually have been for the better! For instance, we now study real-world challenges instead of academic “toy” problems like ℓₚ robustness. We tried to carefully discuss these alternative views in our work.

Javier Rando @javirandor.com · Feb 10

We identify 3 core challenges that make adversarial ML for LLMs harder to define, harder to solve, and harder to evaluate. We then illustrate these with specific case studies: jailbreaks, un-finetunable models, poisoning, prompt injections, membership inference, and unlearning.

Javier Rando @javirandor.com · Feb 10

Perhaps most telling, unlike for image classifiers, manual attacks outperform automated methods at finding worst-case inputs for LLMs! This challenges our ability to automatically evaluate the worst-case robustness of protections and benchmark progress.

Javier Rando @javirandor.com · Feb 10

Now, the field has shifted to LLMs, where we consider subjective notions of safety, allow for unbounded threat models, and evaluate closed-source systems that constantly change. These changes are hindering our ability to produce meaningful scientific progress.

Javier Rando @javirandor.com · Feb 10

Back in the 🐼 days, we dealt with well-defined tasks: misclassify an image by slightly perturbing pixels within an ℓₚ-ball. Also, attack success and defense utility could be easily measured with classification accuracy. Simple objectives that we could rigorously benchmark.

Javier Rando @javirandor.com · Feb 10

Adversarial ML research is evolving, but not necessarily for the better. In our new paper, we argue that LLMs have made problems harder to solve, and even tougher to evaluate. Here’s why another decade of work might still leave us without meaningful progress. 👇

Javier Rando @javirandor.com · Jan 20

Looking forward to this presentation. You can add it to your calendar here cohere.com/events/coher...

Cohere For AI - Javier Rando, AI Safety PhD Student at ETH Zürich

Javier Rando, AI Safety PhD Student at ETH Zürich - Poisoned Training Data Can Compromise LLMs

Javier Rando @javirandor.com · Jan 20

Recently, we have demonstrated that small amounts of poisoned data posted online could compromise large-scale pretraining with backdoors that persist even after alignment arxiv.org/abs/2410.13722

Persistent Pre-Training Poisoning of LLMs

Large language models are pre-trained on uncurated text datasets consisting of trillions of tokens scraped from the Web. Prior work has shown that: (1) web-scraped pre-training datasets can be practic...

Javier Rando @javirandor.com · Jan 20

We poisoned RLHF to introduce backdoors in LLMs that allowed adversaries to elicit harmful generations easily arxiv.org/abs/2311.14455

Universal Jailbreak Backdoors from Poisoned Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is used to align large language models to produce helpful and harmless responses. Yet, prior work showed these models can be jailbroken by finding adv...

Javier Rando @javirandor.com · Jan 20

This Thursday, I will be presenting my work on poisoning RLHF and LLM pretraining @cohereforai.bsky.social

More info here cohere.com/events/coher...

Cohere For AI - Javier Rando, AI Safety PhD Student at ETH Zürich

Javier Rando, AI Safety PhD Student at ETH Zürich - Poisoned Training Data Can Compromise LLMs

Reposted by Javier Rando

Daniel Paleka @dpaleka.bsky.social · Jan 11

Recent LLM forecasters are getting better at predicting the future. But there's a challenge: How can we evaluate and compare AI forecasters without waiting years to see which predictions were right? (1/11)

Javier Rando @javirandor.com · Dec 14

Tomorrow @jakublucki.bsky.social will be presenting the BEST TECHNICAL PAPER at the SoLaR workshop at NeurIPS. Come check our poster and his oral presentation!

Jakub Łucki @jakublucki.bsky.social · Dec 6

Our paper on how unlearning fails to remove hazardous knowledge from LLM weights received 🏆 Best Paper 🏆 award at SoLaR @ NeurIPS!

Join my oral presentation on Saturday at 4:30 pm to learn more.

Reposted by Javier Rando

Kristina Nikolić @nkristina.bsky.social · Dec 12

I am at NeurIPS 🇨🇦, please reach out if you want to grab a coffee!

Reposted by Javier Rando

Michael Aerni @aemai.bsky.social · Dec 10

I am in beautiful Vancouver for #NeurIPS2024 with those amazing folks!
Say hi if you want to chat about ML privacy and security
(or speciality ☕)

Javier Rando @javirandor.com · Dec 10

SPY Lab is in Vancouver for NeurIPS! Come say hi if you see us around 🕵️

Javier Rando @javirandor.com · Dec 10

From left to right the amazing @nkristina.bsky.social @jiezhang-ethz.bsky.social @edebenedetti.bsky.social @javirandor.com @aemai.bsky.social and @dpaleka.bsky.social!

We work on AI Security/Safety/Privacy. Find out more about work in our lab website spylab.ai

We are a research group at ETH Zürich studying how to build secure and private AI.

Javier Rando @javirandor.com · Dec 10

SPY Lab is in Vancouver for NeurIPS! Come say hi if you see us around 🕵️

Javier Rando @javirandor.com · Dec 9

Check out all the details in the offical website llmailinject.azurewebsites.net

llmailinject.azurewebsites.net

Javier Rando @javirandor.com · Dec 9

A new competition on LLM-agents prompt injection is out! Send malicious emails and get agents to perform unauthorised actions.

The competition is hosted at SaTML 2025 and has a pool of $10k in prizes! What are you waiting for?

Santiago Zanella-Beguelin @xefffffff.bsky.social · Dec 9

📢Have experience jailbreaking LLMs?
Want to learn how an indirect / cross prompt injection attack works? Want to try something different to an advent of code?
Then, I have a challenge for you!

The LLMail-Inject competition (llmailinject.azurewebsites.net) starts at 11am UTC (that's in 5min!)

Javier Rando @javirandor.com · Dec 9

2) An Adversarial Perspective on Machine Unlearning for AI Safety

🏆 Best paper award
@solarneurips

📅 Sat 14 Dec. Poster at 11am and Talk in the afternoon.
📍 Room West Meeting 121,122

Paper: arxiv.org/abs/2409.18025

An Adversarial Perspective on Machine Unlearning for AI Safety

Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed. Unlearning methods aim at completely removing hazardous capabilities fro...

Javier Rando @javirandor.com · Dec 9

1) Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition.

📅 Fri 13 Dec 4:30 p.m. PST — 7:30 p.m. PST
📍 Spotlight Poster #5203 (West Ballroom A-D)

arxiv.org/abs/2406.07954

Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

Large language model systems face important security risks from maliciously crafted messages that aim to overwrite the system's original instructions or leak private data. To study this problem, we or...

Javier Rando @javirandor.com · Dec 9

I will be at #NeurIPS2024 in Vancouver. I am excited to meet people working on AI Safety and Security. Drop a DM if you want to meet.

I will be presenting two (spotlight!) works. Come say hi to our posters.

Reposted by Javier Rando

Jakub Łucki @jakublucki.bsky.social · Dec 6

🚨Unlearned hazardous knowledge can be retrieved from LLMs 🚨

Our results show that current unlearning methods for AI safety only obfuscate dangerous knowledge, just like standard safety training.

Here's what we found👇