Gotcha. Yeah, if you're willing to spend the extra training $$$, ALiBi is supposed to length generalize much better, although I didn't verify that.

With new flex_attention baseline, ALiBi is actually slightly faster wall clock, but loss is considerably higher.

What's your plan for eliminating tokenization? Hierarchical architecture seems to be bare minimum. And I suspect some form of latent loss, so that rewording isn't heavily penalized.

ALiBi definitely seems nicer than RoPE, but it doesn't perform as well. Not in nanoGPT speed run, using the per-head-m factors from the paper, anyways.