Kazuki Fujii
@kazukifujii.bsky.social
Tokyo Tech CS Master (Rio Yokota Lab → Jun Sakma Lab) Distributed Training, Sytems for Machine Learning
👉 Read more: arxiv.org/abs/2411.08719
Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
Large Language Models (LLMs) have attracted significant attention due to their human-like language understanding and generation capabilities, as well as their applicability across various domains.
T...
arxiv.org
November 25, 2024 at 12:44 AM
👉 Read more: arxiv.org/abs/2411.08719
We've also observed that the impact of FP8 varies depending on model size and training scenarios (e.g., continual pre-training, from-scratch training, SFT). A comprehensive evaluation requires significant computational resources—this is not a trivial issue. (2/n)
November 25, 2024 at 12:44 AM
We've also observed that the impact of FP8 varies depending on model size and training scenarios (e.g., continual pre-training, from-scratch training, SFT). A comprehensive evaluation requires significant computational resources—this is not a trivial issue. (2/n)