This is related to model calibration, or how confident a llm will tell you that its own answer is correct.
en.wikipedia.org/wiki/Calibra...
It is being actively studied in domains such as code and forecasting: arxiv.org/abs/2401.13835