Research

Spin Glass Model of In-Context Learning

Mapped in-context learning in a linear attention model to a spin glass with real-valued spins; solved the ground state, energy landscape and phase behavior to show how task diversity drives a unique solution that enables in-context prediction in pre-trained transformers.

• Preprint: arXiv:2408.02288
• Published: Phys. Rev. E 112, L013301
• Presented orally at the 29th International Congress on Statistical Physics (StatPhys29).
• Detailed introduction in Chinese: my thesis.