They probably save the spoilers until the very end.

I’m not read the all part, but I believe we can take something from data curation part.

The approach resembles MCTS decoding in a sense that the token which contributes the most for success of the task is likely to choose. However, their approach is more direct: the optimal path is learned by the token probability itself. No need for additional search computation in inference time.

After finding pivotal token, they created synthetic preference dataset which includes (context + accepted token) with positives and (context + rejected token) as negatives.

Especially, they used Pivotal Token Search (PTS) as their post training. In PTS, the model focus of the specific token where it contributes/degenerate success rate by a large margin. The found token is called “pivotal token”.

Marco-o1のMCTSちゃんと理解できてなかったのだけど、本来1) 生成モデル、2) 評価関数モデル、3) 報酬モデルの3つが必要だけど、2-3を1のモデルの対数確率で代替しているっぽい。

もしこの簡易アーキテクチャで元のMCTSと同等の性能が出せるならかなり画期的と言える。

後で裏どりする。

以下のブログに詳細な解説がある。

note.com/hatti8/n/nf8...

Translate: a Japanese contributor published a third party code of MCTS decoder for transformer.
This decode method was used in OpenAI’s o1-like model: Marco-o1.

github.com/Hajime-Y/rea...