DMFT Method of Random Recurrent Neural Networks

We consider a stochastic dynamical system defined by the following equation:

where is also called the colored noise. The activation function is typically chosen as a non-linear function such as the hyperbolic tangent, i.e., . The coupling represents the synaptic weights between neurons, which are assumed to follow a Gaussian distribution , where is a scaling parameter that controls the strength of the connections.

This is a typical random recurrent neural network (RNN) model. Here, we introduce the analysis of this model conducted by H. Sompolinsky et al. in 1988, which is based on the dynamical mean field theory (DMFT).

In the large limit 1 and the long-time limit 2, one can calculate the mean and the correlation of the colored noise as follows:

Using the statistical property , and the definition of the correlation function

one can further simplify Eq. (3) to

On the other hand, we consider the autocorrelation function of the local current , which is defined as

Perform the Fourier transform on Eq. (1), using the definition

and the differentiation property of Fourier transform , we have

Multiplying the above two equations and taking the average, yields

Taking the inverse Fourier transform, the left side of the equation can be calculated as

and the right side can be calculated in a similar way

Then we obtain a differential equation for

In addition, this result can also be obtained by directly calculating the second derivative of .

The fist derivative is

The second derivative is

In the second step, we used the time translation invariance of the system

and in the fourth step, we used the inverse time translation.

The form of the Eq. (16) is similar to the equation of motion of a particle in a potential field, which inspired us to rewrite it as

In order to determine the potential function , we need to express the correlation function as

And then using the Price’s theorem 3, we have

where is the integral of the activation function, and is the variance of at .

This equation describes the dynamics of the correlation function , which can be interpreted as the motion of a particle in the potential field . The shape of the potential function determines the stability and the dynamics of the system.


Footnotes

  1. The central limit theorem can be applied. ↩

  2. According to the ergodic hypothesis, it can be concluded that the time-average is equal to the ensemble-average. ↩

  3. One formulation of Price’s theorem is:
    Consider two Gaussian random variables and with covariance , satisfying the joint probability distribution . For any function , define its expectation

    then we have
    ↩

# DMFT # RNN