Done.

I wrote a blog post about it.

Link: medium.com/@kimon.fount...

2) Can a neural network discover instructions for performing multiplication itself?

The answer to the first question is yes, with high probability and up to some arbitrary, predetermined precision (see the quoted post).

Link to the paper: arxiv.org/abs/2502.16763
Link to the repository: github.com/opallab/bina...

Learning to execute arithmetic exactly, with high probability, can be quite expensive. In the plot, 'ensemble complexity' refers to the number of independently trained models required to achieve exact learning with high probability. ell is the number of bits per number in the input.

New paper: Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networks

Learning to execute arithmetic exactly, with high probability, can be quite expensive. In the plot, 'ensemble complexity' refers to the number of independently trained models required to achieve exact learning with high probability. ell is the number of bits per number in the input.

I never understood the point of trams. They're slow and expensive. I've been to two cities that built them while I was there, Edinburgh and Athens, and in both cases, the projects were born out of corruption. Especially in Edinburgh, it was a disaster. en.wikipedia.org/wiki/Edinbur...

Thanks, the connection to formal languages is quite interesting. I have a section in the repo regarding formal languages but it's small mainly because it's not a topic that I am familiar with. I will add them!

Update, 14 empirical papers added!

The SIAM Conference on Optimization 2026 will be in Edinburgh! I don’t really work on optimization anymore (at least not directly), but it’s cool to see a major optimization conference taking place where I did my PhD.

Currently NeurIPS has 21390 submissions. The final number last year was 15671.

Observation made by my student George Giapitzakis.

Got a pin this morning

Einstein problem: en.wikipedia.org/wiki/Einstei...

Our new work on scaling laws that includes compute, model size, and number of samples. The analysis involves an extremely fine-grained analysis of online sgd built up over the last 8 years of understanding sgd on simple toy models (tensors, single index models, multi index model)

Hey, I definitely predicted this correctly.

ChatGPT gives me the ability to expand my search capabilities on topics that I can only roughly describe, or even illustrate with a figure, when I don’t know the exact keywords to use in a Google search.

That's a comprehensive study on the expressivity for parallel algorithms, their in- and out-of-distribution learnability, and it includes a lot of experiments.

link: arxiv.org/abs/2410.01686

Positional Attention is accepted at ICML 2025! Thanks to all co-authors for the hard work (64 pages). If you’d like to read the paper, check the quoted post.

NeurIPS 2026 in the Cyclades. Just saying.

Wait, isn't that America?

This is different from simply replacing the discontinuous activation in the neural network with a continuous one and then using standard NTK.

link: openreview.net/forum?id=kfd...

They analyze a modified gradient flow, where the Jacobian for the training data uses an approximate derivative of the activation function.

I enjoyed reading the paper "A Generalized Neural Tangent Kernel for Surrogate Gradient Learning" (Spotlight, NeurIPS 2024).

They extend the NTK framework to activation functions that have finitely many jumps.