The simplest alternative to LoRA is to use SVD on the model's weight matrices, then fine-tune the singular values directly. Oddly, this is the most recent technique, called SVF, published in the Transformers² paper (arxiv.org/abs/2501.06...).

Are you still using LoRA to fine-tune your LLM? 2024 has seen an explosion of new parameter-efficient fine tuning technique (PEFT), thanks to clever uses of the singular value decomposition (SVD). Let's dive into the alphabet soup: SVF, SVFT, MiLoRA, PiSSA, LoRA-XS 🤯...

The NYT published a bizarre religious article on the Pentium division bug exactly 30 years ago today: "Pentium and Our Crisis of Faith." It argued that you need to have faith in a computer's results, so the Pentium bug was like Martin Luther's Protestant revolt.

www.nytimes.com/1994/12/28/o...

Exploring the full bluesky firehose, in three dimensions: firehose3d.theo.io

The current public Jetstream instance has been running with `zstd` compression support for ~14 hours now, checking in on performance.

The instance supports ~12 consumers right now with varying filters, but that fluctuates throughout the day. All consumers are currently caught up with live.