Spin Glass Model of In-Context Learning

Mapped in-context learning in a linear attention model to a spin glass with real-valued spins; solved the ground state, energy landscape and phase behavior to show how task diversity drives a unique solution that enables in-context prediction in pre-trained transformers.

• Authors: Yuhao Li, Ruoran Bai, Haiping Huang
• Code Repository: GitHub
• Preprint: arXiv:2408.02288
• Published in: Phys. Rev. E 112, L013301 (2025)
• Presented orally at the 29th International Congress on Statistical Physics (Florence, Italy, 2025).
• Presented as a poster at the 8th National Conference on Statistical Physics and Complex Systems (Ningbo, China, 2025).
• Detailed Introduction in Chinese: My Bachelor Thesis.