Stationarity autocovariance and ARMA models

Definition Throughout, dates $t \in \mathbb{Z}$ are integers counting periods. A process $(X_t)_{t \in \mathbb{Z}}$ with $\mathbb{E}[X_t^2] < \infty$ for every $t$ is weakly stationary if its mean is constant and the covariance between two dates depends only on their distance, that is, for all $t, h \in \mathbb{Z}$,

with a constant $\mu$ and a function $\gamma$ that do not depend on $t$. The process is strictly stationary if for every $k$, all dates $t_1, \dots, t_k$ and every shift $s$, the vector $(X_{t_1 + s}, \dots, X_{t_k + s})$ has the same joint distribution as $(X_{t_1}, \dots, X_{t_k})$. Strict stationarity together with finite second moments implies weak stationarity. The converse fails in general, but for a Gaussian process, whose finite-dimensional distributions are multivariate normal and therefore fixed by means and covariances, the two notions coincide. A strictly stationary process need not be weakly stationary, since its second moments may be infinite, as for independent standard Cauchy variables.

Definition For a weakly stationary process, the autocovariance function assigns to each lag $h \in \mathbb{Z}$ the covariance

the same for every $t$. It satisfies $\gamma(0) = \operatorname{Var}(X_t)$, $\gamma(-h) = \gamma(h)$ and $\lvert \gamma(h) \rvert \le \gamma(0)$ by the Cauchy-Schwarz inequality. It is positive semidefinite, meaning $\sum_{i,j} a_i a_j \gamma(t_i - t_j) \ge 0$ for all real $a_i$ and dates $t_i$, because this sum is the variance of $\sum_i a_i X_{t_i}$. When $\gamma(0) > 0$, the autocorrelation function is $\rho(h) = \gamma(h)/\gamma(0)$, with $\rho(0) = 1$ and $\lvert \rho(h) \rvert \le 1$. From one observed path $x_1, \dots, x_n$, the sample autocovariance

for $0 \le h < n$, is an estimate, and statements about its sampling error need further assumptions on the dependence of the process. Autocovariance compares one series with itself at different dates. It is not the covariance between two assets at the same date.

Definition An AR(1) process with coefficient $\phi \in \mathbb{R}$ satisfies, for every date $t \in \mathbb{Z}$,

where $(\varepsilon_t)$ is white noise with variance $\sigma_\varepsilon^2 > 0$. A solution is causal if each $X_t$ depends only on the current and past shocks $\varepsilon_s$, $s \le t$. If $\lvert \phi \rvert < 1$, the causal stationary representation

converges in mean square and is the unique weakly stationary solution. To derive it, iterate the recursion $k$ times,

For a weakly stationary solution the first term has second moment $\phi^{2k}\,\mathbb{E}[X_{t-k}^2]$, a constant times $\phi^{2k}$, which tends to zero. The partial sums converge in mean square because the shocks are uncorrelated, so a block of terms from $j = m$ to $j = k$ has second moment $\sigma_\varepsilon^2\sum_{j=m}^{k}\phi^{2j}$, which tends to zero as $m$ grows. The limit has mean zero and

The variance is $\sigma_\varepsilon^2\sum_j \phi^{2j}$. For $h \ge 1$ the iterated recursion writes $X_t$ as $\phi^h X_{t-h}$ plus the shocks dated after $t - h$, which are uncorrelated with $X_{t-h}$, so $\operatorname{Cov}(X_t, X_{t-h}) = \phi^h\gamma(0)$. Only second moments are used, and independent shocks are needed only for statements about distributions. If $\lvert \phi \rvert > 1$, a stationary solution still exists, but it is a series in future shocks and so is not causal. If $\lvert \phi \rvert = 1$, no stationary solution exists. A process with mean $\mu$ follows $X_t - \mu = \phi(X_{t-1} - \mu) + \varepsilon_t$.

Definition Let $L$ be the lag operator, $LX_t = X_{t-1}$, and define the autoregressive and moving-average polynomials

Here $\phi(z)$ and $\theta(z)$ are polynomials, and for $p = 1$ the coefficient $\phi_1$ is the $\phi$ of the AR(1) model. A weakly stationary process satisfies an ARMA($p$, $q$) model if $\phi(L)X_t = \theta(L)\varepsilon_t$ for white noise $(\varepsilon_t)$, that is,

The moving-average part uses a plus sign, and software conventions differ, so the sign must be checked. Assume that $\phi(z)$ and $\theta(z)$ have no common zeros, after any common factors are canceled. The following results are stated without proof. A weakly stationary solution exists, and is then unique, exactly when $\phi(z) \ne 0$ for every complex $z$ with $\lvert z \rvert = 1$. It is causal, $X_t = \sum_{j \ge 0}\psi_j\varepsilon_{t-j}$ with $\sum_j \lvert \psi_j \rvert < \infty$ and $\sum_j \psi_j z^j = \theta(z)/\phi(z)$, exactly when $\phi(z) \ne 0$ for all $\lvert z \rvert \le 1$. It is invertible, meaning $\varepsilon_t = \sum_{j \ge 0}\pi_j X_{t-j}$ with $\sum_j \lvert \pi_j \rvert < \infty$, exactly when $\theta(z) \ne 0$ for all $\lvert z \rvert \le 1$. For $\phi \ne 0$ the only root of $1 - \phi z$ is $1/\phi$, which lies outside the unit circle exactly when $\lvert \phi \rvert < 1$. A finite moving average is weakly stationary for any coefficients, invertible or not.

Definition Let $\mathcal{F}_t$ be the information generated by $X_s$, $s \le t$. For a causal stationary AR(1) process with $\lvert \phi \rvert < 1$, assume beyond white noise that for every $t$

By the tower property the future shocks then have conditional mean $0$, conditional variance $\sigma_\varepsilon^2$ and zero conditional covariances given $\mathcal{F}_t$. Writing

for a horizon $h \ge 1$ gives the conditional mean and the conditional forecast variance

When $\phi \ne 0$ the conditional variance is strictly below $\gamma(0)$ for every $h$ and increases to $\gamma(0)$ as $h \to \infty$, while the forecast tends to the mean $0$. When $\phi = 0$ the past carries no information, and every forecast is $0$ with variance $\gamma(0) = \sigma_\varepsilon^2$. Uncorrelated white noise alone does not give these conditional statements. It still makes $\phi^h X_t$ the best linear predictor of $X_{t+h}$ from $X_s$, $s \le t$, with mean squared error given by the same sum.

Definition An autoregressive model has a unit root when its polynomial has the root $z = 1$, that is, $\phi(1) = 0$, or equivalently $\phi_1 + \dots + \phi_p = 1$. The simplest case is the random walk

the AR(1) recursion with $\phi = 1$. Started at $X_0 = 0$, it equals $X_t = \sum_{s=1}^{t}\varepsilon_s$ for $t \ge 1$, so with white noise shocks $\operatorname{Var}(X_t) = t\,\sigma_\varepsilon^2$, which grows with $t$, and the process is not weakly stationary. No stationary solution exists at all. For any solution, $X_t - X_{t-n}$ is a sum of $n$ uncorrelated shocks with variance $n\sigma_\varepsilon^2$, while for a weakly stationary process $\operatorname{Var}(X_t - X_{t-n}) \le 4\gamma(0)$ for every $n$. The formula $\sigma_\varepsilon^2/(1 - \phi^2)$ cannot be used at $\phi = 1$, because its derivation needed $\lvert \phi \rvert < 1$. The first difference $\Delta X_t = X_t - X_{t-1} = \varepsilon_t$ is stationary, and a process that is not stationary but whose first difference is a stationary ARMA process is called integrated of order one. Under the conditional assumptions of the previous card, the random walk forecast is $\mathbb{E}[X_{t+h} \mid \mathcal{F}_t] = X_t$ with conditional variance $h\,\sigma_\varepsilon^2$, which grows without bound in $h$. Deciding from data whether a root equals one is a statistical testing problem outside this lesson.