Scaling of Entropy Fluctuations and Complexity Across Free-Energy Levels

This problem was an assignment in the ‘Statistical Mechanics of Neural Networks’ course at SYSU, Fall 2024. In this post, we analyze how the scaling of entropy fluctuations affects the configurational entropy (complexity) across different free-energy levels in a simple toy model.

Question

We consider a simple toy model in which the energy and entropy fluctuations scale differently with system size. Specifically, let

and study how the scaling exponent affects the number of configurations at different free-energy levels.

Analysis

The free energy is defined as , so the probability distribution of can be written as

For the energy term, we have

Similarly, for the entropy term,

Putting everything together, we obtain

We now consider a system of Ising spins, which has a total of configurations. The expected number of configurations with free energy is therefore

By analogy with thermodynamic entropy, we define the configurational entropy (also called complexity in the context of one-step replica symmetry breaking) for the free-energy density as

Evaluating the large- limit,

The resulting configurational entropy takes the form

which reveals a sharp transition at , separating regimes in which entropy fluctuations are either subdominant or dominant compared to energy fluctuations.

Numerical Illustrations

Left: Configurational entropy as a function of temperature for different values of .
The free-energy density is fixed to . For , the complexity remains at . For , the complexity vanishes. At the marginal case , exhibits a nontrivial temperature dependence.

Right: Configurational entropy as a function of for fixed temperature.
The parameters are chosen as and . As , the configurational entropy exhibits a discontinuous jump at .