Linear latent variable models: the lava-package

Abstract

An R package for specifying and estimating linear latent variable models is presented. The philosophy of the implementation is to separate the model specification from the actual data, which leads to a dynamic and easy way of modeling complex hierarchical structures. Several advanced features are implemented including robust standard errors for clustered correlated data, multigroup analyses, non-linear parameter constraints, inference with incomplete data, maximum likelihood estimation with censored and binary observations, and instrumental variable estimators. In addition an extensive simulation interface covering a broad range of non-linear generalized structural equation models is described. The model and software are demonstrated in data of measurements of the serotonin transporter in the human brain.

Appendices

Appendix A: Some zero-one matrices

In this section we will define a few matrix-operators in order to define various conditional moments. Let $\varvec{B}\in \mathbb R ^{n\times m}$ be a matrix, and define the indices $\varvec{x} = \{x_{1},\ldots ,x_{k}\}\in \{1,\ldots ,n\}$, and $\varvec{y} = \{y_{1},\ldots ,y_{l}\}\in \{1,\ldots ,m\}$. We define $\varvec{J}_{n,\varvec{x}} = \varvec{J}_{\varvec{x}} \in \mathbb R ^{(n-k)\times n}$ as the $n\times n$ identify matrix with rows ${\varvec{x}}$ removed. E.g.

$$\begin{aligned} {\varvec{J}}_{6,(3,4)} = \left(\begin{array}{l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l} 1&0&0&0&0&0 \\ 0&1&0&0&0&0 \\ 0&0&0&0&1&0 \\ 0&0&0&0&0&1 \\ \end{array}\right). \end{aligned}$$

To remove rows $\varvec{x}$ from $\varvec{B}$ we simply multiply from the left with $\varvec{J}_{n,\varvec{x}}$. If we in addition want to remove columns $\varvec{y}$ we multiply with the transpose of $\varvec{J}_{n,\varvec{y}}$ from the right:

$$\begin{aligned} \varvec{J}_{n,\varvec{x}}\varvec{B}\varvec{J}_{n,\varvec{y}}^{\prime }. \end{aligned}$$

(85)

We will use the notation $\varvec{J}$ to denote the matrix that removes all latent variables ($\varvec{\eta }$) from the vector of all variable, $\varvec{U}$ as defined in (8). We denote $\varvec{J}_{\varvec{Y}}$ the matrix that only keeps endogenous variables ($\varvec{Y}$).

We also need an operator that cancels out rows or columns of a matrix/vector. Define the square matrix $\varvec{p}_{n,\varvec{x}}\in \mathbb R ^{n\times n}$ as the identity-matrix with diagonal elements at position $\varvec{x}$ canceled out:

$$\begin{aligned} \varvec{p}_{n,\varvec{x}}(i,j) = {\left\{ \begin{array}{ll} 1,&i=j, \ i\not \in \varvec{x}, \\ 0,&\text{ otherwise}. \end{array}\right.} \end{aligned}$$

(86)

e.g.

$$\begin{aligned} \varvec{p}_{6,(3,4)} = \left(\begin{array}{l@{\quad }l@{\quad }l@{\quad }l@{\quad }l@{\quad }l} 1&0&0&0&0&0 \\ 0&1&0&0&0&0 \\ 0&0&0&0&0&0 \\ 0&0&0&0&0&0 \\ 0&0&0&0&1&0 \\ 0&0&0&0&0&1 \\ \end{array}\right). \end{aligned}$$

To cancel out rows $\varvec{x}$ and columns $\varvec{y}$ of the matrix $\varvec{B}\in \mathbb R ^{n\times m}$ we calculate

$$\begin{aligned} \varvec{p}_{n,\varvec{x}}\varvec{B}\varvec{p}_{m,\varvec{y}}^{\prime }. \end{aligned}$$

We will use $\varvec{p}_{\varvec{X}}$ and $\varvec{p}_{\complement \varvec{X}}$ as the matrix that cancels out the rows corresponding to the index of the exogenous variables ($\varvec{X}$) respectively the matrix that cancels out all rows but the ones corresponding to $\varvec{X}$.

Appendix B: The score function and information

In this section we will calculate the analytical derivatives of the log-likelihood. In order to obtain these results we will first introduce the notation of some common matrix operations. Let $\varvec{A}\in \mathbb R ^{m\times n}$, then we define the column-stacking operation:

$$\begin{aligned} \,\text{ vec}\,(\varvec{A}) = \begin{pmatrix} a_1 \\ \vdots \\ a_n \end{pmatrix}, \end{aligned}$$

where $a_i$ denotes the ith column of $\varvec{A}$. The unique commutation matrix, $\mathbb R ^{mn\times mn}$ is defined by

$$\begin{aligned} \varvec{K}^{(m,n)}\,\text{ vec}\,(\varvec{A}) = \,\text{ vec}\,(\varvec{A}^{\prime }). \end{aligned}$$

(87)

Letting $\varvec{H}^{(i,j)}$ be the $m\times n$-matrix with one at position $(i,j)$ and zero elsewhere, then

$$\begin{aligned} \varvec{K}^{(m,n)} = \sum _{i=1}^m\sum _{j=1}^n (\varvec{H}^{(i,j)}\otimes \varvec{H}^{(i,j)}{}^{\prime }), \end{aligned}$$

e.g.

It should be noted that product with a commutation matrix can be implemented very efficiently instead of relying on a direct implementation of the above mathematical definition.

Let $\varvec{A}\in \mathbb R ^{m\times n}$ and $\varvec{B}\in \mathbb R ^{p\times q}$ then the Kronecker product is the $mp\times nq$-matrix:

$$\begin{aligned} \varvec{A}\otimes \varvec{B} = \begin{pmatrix} a_{11}\varvec{B}&\cdots&a_{1n}\varvec{B} \\ \vdots&\vdots \\ a_{m1}\varvec{B}&\cdots&a_{mn}\varvec{B} \\ \end{pmatrix} \end{aligned}$$

We will calculate the derivatives of (20) by means of matrix differential calculus. The Jacobian matrix of a matrix-function $F:\mathbb R ^n\rightarrow \mathbb R ^{m\times p}$ is the $mp\times n$ matrix defined by

$$\begin{aligned} DF(\varvec{\theta }) = \frac{\partial \,\text{ vec}\,F(\varvec{\theta })}{\partial \varvec{\theta }^{\prime }}. \end{aligned}$$

Letting $\,\text{ d}\!\,$ denote the differential operator (see Magnus and Neudecker 1988), the first identification rule states that $\,\text{ d}\!\,\,\text{ vec}\,F(\varvec{\theta }) = A(\varvec{\theta })\,\text{ d}\!\,\varvec{\theta } \Rightarrow DF(\varvec{\theta }) = A(\varvec{\theta })$.

1.1 Appendix 14.1: Score function

Using the identities $\,\text{ d}\!\,\log \left|\varvec{X}\right| = \,\text{ tr}\,(\varvec{X}^{-1}\,\text{ d}\!\,\varvec{X})$ and $\,\text{ d}\!\,\varvec{X}^{-1} = -\varvec{X}^{-1}(\,\text{ d}\!\,\varvec{X})\varvec{X}^{-1}$, and applying the product rule we get

$$\begin{aligned} \,\text{ d}\!\,\ell (\varvec{\theta })&= -\frac{n}{2}\,\text{ tr}\,({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}) - \frac{n}{2}\,\text{ tr}\,(\,\text{ d}\!\,\{\widehat{\varvec{\varSigma }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\}) \end{aligned}$$

(88)

$$\begin{aligned}&= - \frac{n}{2}\,\text{ tr}\,({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}) + \frac{n}{2}\,\text{ tr}\,(\widehat{\varvec{\varSigma }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}[\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}]{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}), \end{aligned}$$

(89)

where

$$\begin{aligned} \,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}&= \left\{ \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}\right\} {\varvec{P}}_{\varvec{\theta }}{\varvec{G}}_{\varvec{\theta }}^{\prime } + {\varvec{G}}_{\varvec{\theta }}\left\{ \,\text{ d}\!\,{\varvec{P}}_{\varvec{\theta }}{\varvec{G}}_{\varvec{\theta }}^{\prime }\right\} \end{aligned}$$

(90)

$$\begin{aligned}&= \left\{ \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}\right\} {\varvec{P}}_{\varvec{\theta }}{\varvec{G}}_{\varvec{\theta }}^{\prime } + {\varvec{G}}_{\varvec{\theta }}\left\{ \,\text{ d}\!\,{\varvec{P}}_{\varvec{\theta }}\right\} {\varvec{G}}_{\varvec{\theta }}^{\prime } + {\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}\left\{ \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}\right\} ^{\prime } \end{aligned}$$

(91)

$$\begin{aligned}&= \left\{ \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}\right\} {\varvec{P}}_{\varvec{\theta }}{\varvec{G}}_{\varvec{\theta }}^{\prime } + [\left\{ \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}\right\} {\varvec{P}}_{\varvec{\theta }}{\varvec{G}}_{\varvec{\theta }}^{\prime }]^{\prime } + {\varvec{G}}_{\varvec{\theta }}\left\{ \,\text{ d}\!\,{\varvec{P}}_{\varvec{\theta }}\right\} {\varvec{G}}_{\varvec{\theta }}^{\prime }, \end{aligned}$$

(92)

and

$$\begin{aligned} \,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}= \varvec{J}(\varvec{1}_m-{\varvec{A}}_{\varvec{\theta }})^{-1}\left\{ \,\text{ d}\!\,{\varvec{A}}_{\varvec{\theta }}\right\} (\varvec{1}_m-{\varvec{A}}_{\varvec{\theta }})^{-1}. \end{aligned}$$

(93)

Taking $\,\text{ vec}\,$’s it follows that

$$\begin{aligned} \,\text{ d}\!\,\,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}= \left[((\varvec{1}_m-{\varvec{A}}_{\varvec{\theta }})^{-1})^{\prime }\otimes {\varvec{G}}_{\varvec{\theta }}\right]\,\text{ d}\!\,\,\text{ vec}\,{\varvec{A}}_{\varvec{\theta }}\,\text{ d}\!\,\varvec{\theta }, \end{aligned}$$

(94)

hence by the first identification rule

$$\begin{aligned} \frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} = \left[((\varvec{1}_m-{\varvec{A}}_{\varvec{\theta }})^{-1})^{\prime }\otimes {\varvec{G}}_{\varvec{\theta }}\right] \frac{\partial \,\text{ vec}\,{\varvec{A}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}, \end{aligned}$$

(95)

and similarly

$$\begin{aligned} \frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} = (\varvec{1}_{k^2} + \varvec{K}^{(k,k)}) \left[{\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}\otimes \varvec{1}_k\right] \frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}+ \left[{\varvec{G}}_{\varvec{\theta }}\otimes {\varvec{G}}_{\varvec{\theta }}\right]\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }},\quad \end{aligned}$$

(96)

and finally (exploiting the symmetry of ${\varvec{\varOmega }}_{\varvec{\theta }}$ and commutative property under the trace operator) we obtain the gradient

$$\begin{aligned} \frac{\partial \ell (\varvec{\theta })}{\partial \varvec{\theta }} = \frac{n}{2}\left(\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right)^{\prime }\,\text{ vec}\,\left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\widehat{\varvec{\varSigma }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right] -\frac{n}{2}\left(\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right)^{\prime }\,\text{ vec}\,\left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right].\qquad \end{aligned}$$

(97)

Next we examine the model including a mean structure (20). W.r.t. to the first differential we observe that

$$\begin{aligned} \,\text{ d}\!\,\,\text{ tr}\,\left\{ {\varvec{T}}_{\varvec{\theta }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right\} = -\,\text{ tr}\,\left\{ {\varvec{T}}_{\varvec{\theta }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}(\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}){\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right\} + \,\text{ tr}\,\left\{ (\,\text{ d}\!\,{\varvec{T}}_{\varvec{\theta }}){\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right\} . \end{aligned}$$

(98)

Hence

$$\begin{aligned} \frac{\partial \ell (\varvec{\theta })}{\partial \varvec{\theta }}&= \frac{n}{2}\left(\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right)^{\prime }\,\text{ vec}\,\left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{T}}_{\varvec{\theta }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right] -\frac{n}{2}\left(\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right)^{\prime }\,\text{ vec}\,\left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right]\nonumber \\&\quad - \frac{n}{2}\left(\frac{\partial \,\text{ vec}\,{\varvec{T}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right)^{\prime }\,\text{ vec}\,\left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right). \end{aligned}$$

(99)

Further by the chain-rule

$$\begin{aligned} \frac{\partial \,\text{ vec}\,{\varvec{T}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} = \frac{\partial (\widehat{\varvec{\mu }}-{\varvec{v}}_{\varvec{\theta }})}{\partial \varvec{\theta }^{\prime }} \frac{\partial \,\text{ vec}\,{\varvec{T}}_{\varvec{\theta }}}{\partial (\widehat{\varvec{\mu }}-{\varvec{\xi }}_{\varvec{\theta }})^{\prime }} = - \left[\varvec{1}_k\otimes (\widehat{\varvec{\mu }}-{\varvec{\xi }}_{\varvec{\theta }}) + (\widehat{\varvec{\mu }}-{\varvec{\xi }}_{\varvec{\theta }})\otimes \varvec{1}_k\right]\frac{\partial {\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }},\nonumber \\ \end{aligned}$$

(100)

and

$$\begin{aligned} \,\text{ d}\!\,{\varvec{\xi }}_{\varvec{\theta }} = (\,\text{ d}\!\,{\varvec{G}}_{\varvec{\theta }}){\varvec{v}}_{\varvec{\theta }} + {\varvec{G}}_{\varvec{\theta }}(\,\text{ d}\!\,{\varvec{v}}_{\varvec{\theta }}). \end{aligned}$$

(101)

Taking $\,\text{ vec}\,$ (${\varvec{G}}_{\varvec{\theta }}{\varvec{v}}_{\varvec{\theta }} = \varvec{1}{\varvec{G}}_{\varvec{\theta }}{\varvec{v}}_{\varvec{\theta }}$):

$$\begin{aligned} \frac{\partial {\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} = ({\varvec{v}}_{\varvec{\theta }}^{\prime }\otimes \varvec{1}_k)\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} + {\varvec{G}}_{\varvec{\theta }}\frac{\partial {\varvec{v}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}. \end{aligned}$$

(102)

We have calculated the full score but in some situations it will be useful to evaluate the score in a single point. The contribution of a single observation to the log-likelihood is

$$\begin{aligned} \ell \le (\varvec{\theta }\mid \varvec{z}_i) \propto \frac{1}{2} \log \left|{\varvec{\varOmega }}_{\varvec{\theta }}\right| + \frac{1}{2}(\varvec{z}_i-{\varvec{\xi }}_{\varvec{\theta }})^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}(\varvec{z}_i-{\varvec{\xi }}_{\varvec{\theta }}), \end{aligned}$$

(103)

or as in (20) where we simply exchange ${\varvec{T}}_{\varvec{\theta }}$ with $\varvec{T}_{\varvec{z}_i,\varvec{\theta }} = (\varvec{z}_i-{\varvec{\xi }}_{\varvec{\theta }})(\varvec{z}_i-{\varvec{\xi }}_{\varvec{\theta }})^{\prime }$, hence the score is as in (99) where (100) is calculated with $\varvec{z}_i$ instead of $\widehat{\varvec{\mu }}$. Alternatively, letting $\varvec{z_i}-{\varvec{\xi }}_{\varvec{\theta }} = {\varvec{u}}_{\varvec{\theta }} = {\varvec{u}}_{\varvec{\theta }}(i)$:

$$\begin{aligned} \,\text{ d}\!\,({\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }})&= {\varvec{u}}_{\varvec{\theta }}^{\prime }\left[(2{\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\,\text{ d}\!\,{\varvec{u}}_{\varvec{\theta }} + (\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}){\varvec{u}}_{\varvec{\theta }}\right] \nonumber \\&= - {\varvec{u}}_{\varvec{\theta }}^{\prime }\left[(2{\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\,\text{ d}\!\,{\varvec{\xi }}_{\varvec{\theta }} + {\varvec{\varOmega }}_{\varvec{\theta }}^{-1}(\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}){\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}\right], \end{aligned}$$

(104)

where we used that for constant symmetric $\varvec{A}$ the differential of a quadratic form is

$$\begin{aligned} \,\text{ d}\!\,(\varvec{u}^{\prime }\varvec{A}\varvec{u}) = 2\varvec{u}^{\prime }(\varvec{A})\,\text{ d}\!\,\varvec{u}. \end{aligned}$$

(105)

Hence the contribution to the score function of the $i$th observation is

$$\begin{aligned} \mathcal S _i(\varvec{\theta })&= -\frac{1}{2}\Big \{ \,\text{ vec}\,({\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} -2{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\frac{\partial \,\text{ vec}\,{\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \nonumber \\&\quad - ({\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\otimes {\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \Big \} \nonumber \\&= -\frac{1}{2}\left\{ \left(\,\text{ vec}\,({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}) \!-\! \,\text{ vec}\,({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\right)\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \!-\!2{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\frac{\partial \,\text{ vec}\,{\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right\} ,\nonumber \\ \end{aligned}$$

(106)

where the score-function evaluated in $\varvec{\theta }$ is $\mathcal S (\varvec{\theta }) = \sum _{i=1}^n \mathcal S _i(\varvec{\theta })$.

1.2 The information matrix

The second order partial derivative is given by

$$\begin{aligned} \frac{\partial \ell (\varvec{\theta })}{\partial \theta _i\theta _j} \!=\! \!-\!\frac{1}{2}\frac{\partial }{\partial \theta _i}\left[\! \left\{ \,\text{ vec}\,\left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right) \!-\! \,\text{ vec}\,\left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right) \right\} \frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \theta _j} \!-\! 2{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\frac{\partial {\varvec{\xi }}_{\varvec{\theta }}}{\partial \theta _j} \!\right]\!.\nonumber \\ \end{aligned}$$

(107)

Taking negative expectation with respect to the true parameter $\varvec{\theta }_0$ we obtain the expected information (Magnus and Neudecker 1988), which get rid of all second order derivatives

$$\begin{aligned} \mathcal I (\varvec{\theta }_0)&= \frac{1}{2}\left(\left. \frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right|_{\varvec{\theta }=\varvec{\theta }_0} \right)^{\prime }(\varvec{\varOmega }_{\varvec{\theta }_0}^{-1}\otimes \varvec{\varOmega }_{\varvec{\theta }_0}^{-1}) \left(\left. \frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right|_{\varvec{\theta }=\varvec{\theta }_0} \right) \end{aligned}$$

(108)

$$\begin{aligned}&\quad + \left(\left. \frac{\partial {\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right|_{\varvec{\theta }=\varvec{\theta }_0} \right)^{\prime } \varvec{\varOmega }_{\varvec{\theta }_0}^{-1} \left(\left. \frac{\partial {\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right|_{\varvec{\theta }=\varvec{\theta }_0} \right). \end{aligned}$$

(109)

We will further derive the observed information in the case where the second derivatives vanishes in the case of the matrix functions ${\varvec{A}}_{\varvec{\theta }}, {\varvec{P}}_{\varvec{\theta }}$ and ${\varvec{v}}_{\varvec{\theta }}$. Now

$$\begin{aligned} \,\text{ d}\!\,^2{\varvec{G}}_{\varvec{\theta }} = \,\text{ d}\!\,\left[\varvec{J}(\varvec{1}-{\varvec{A}}_{\varvec{\theta }})^{-1}(\,\text{ d}\!\,{\varvec{A}}_{\varvec{\theta }})(\varvec{1}-{\varvec{A}}_{\varvec{\theta }})^{-1}\right]. \end{aligned}$$

(110)

Hence

$$\begin{aligned} \frac{\partial ^2 {\varvec{G}}_{\varvec{\theta }}}{\partial \theta _i\partial \theta _j} = {\varvec{G}}_{\varvec{\theta }}\left[\frac{\partial {\varvec{A}}_{\varvec{\theta }}}{\partial \theta _i}(\varvec{1}-{\varvec{A}}_{\varvec{\theta }})^{-1}\frac{\partial {\varvec{A}}_{\varvec{\theta }}}{\partial \theta _j} + \frac{\partial {\varvec{A}}_{\varvec{\theta }}}{\partial \theta _j}(\varvec{1}-{\varvec{A}}_{\varvec{\theta }})^{-1}\frac{\partial {\varvec{A}}_{\varvec{\theta }}}{\partial \theta _i}\right](\varvec{1}-{\varvec{A}}_{\varvec{\theta }})^{-1}.\nonumber \\ \end{aligned}$$

(111)

Next we will find the derivative of (96). We let $m$ denote the number of variables, $p$ the number of parameters, and $k$ the number of observed variable (e.g. ${\varvec{G}}_{\varvec{\theta }}\in \mathbb R ^{k\times m}$ and the number of columns in the derivatives are $p$). We have ${\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}\in \mathbb R ^{k\times m}$ and using rules for evaluating the differential of Kronecker-product (see Magnus and Neudecker 1988, pp. 184) we obtain

$$\begin{aligned} \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }^{\prime }}( {\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}\otimes \varvec{1}_{k})&= \left(\varvec{1}_{m}\otimes \varvec{K}^{(k,k)}\otimes \varvec{1}_k\right) \left(\varvec{1}_{km}\otimes \,\text{ vec}\,\varvec{1}_k\right) \frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \nonumber \\&= \left(\varvec{1}_{m}\otimes \varvec{K}^{(k,k)}\otimes \varvec{1}_k\right)\left(\varvec{1}_{km} \otimes \,\text{ vec}\,\varvec{1}_k\right)\nonumber \\&\left[ ({\varvec{P}}_{\varvec{\theta }}\otimes \varvec{1}_k)\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} + (\varvec{1}_m\otimes {\varvec{G}}_{\varvec{\theta }})\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right]. \qquad \end{aligned}$$

(112)

And

$$\begin{aligned} \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }}\left[({\varvec{G}}_{\varvec{\theta }}\otimes {\varvec{G}}_{\varvec{\theta }})\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right]&= \left[\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}^{\prime } \otimes \varvec{1}_{k^2}\right]\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}\otimes {\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \nonumber \\&= \left[\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}^{\prime }\otimes \varvec{1}_{k^2}\right] \left(\varvec{1}_m\otimes \varvec{K}^{(m,k)}\otimes \varvec{1}_k\right)\\&\times \left(\varvec{1}_{km}\otimes \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}+ \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}\otimes \varvec{1}_{km}\right) \frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}. \nonumber \end{aligned}$$

(113)

Hence from (112) and (113) and using rules for applying the $\,\text{ vec}\,$ operator on products of matrices we obtain

$$\begin{aligned} \frac{\partial ^2\,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }\partial \varvec{\theta }^{\prime }}&= \left[\left(\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right)^{\prime }\otimes \left( \varvec{1}_{k^2}+\varvec{K}^{(k,k)} \right)\right] \left(\varvec{1}_{m}\otimes \varvec{K}^{k+k}\otimes \varvec{1}_k\right)\left(\varvec{1}_{km}\otimes \,\text{ vec}\,\varvec{1}_k\right) \nonumber \\&\times \left[({\varvec{P}}_{\varvec{\theta }}\otimes \varvec{1}_k)\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} + (\varvec{1}_m\otimes {\varvec{G}}_{\varvec{\theta }})\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right] \nonumber \\&+\Big (\varvec{1}_p \otimes ({\varvec{G}}_{\varvec{\theta }}{\varvec{P}}_{\varvec{\theta }}\otimes \varvec{1}_k)\Big ) \frac{\partial ^2\,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }\partial \varvec{\theta }^{\prime }} +\left(\left(\frac{\partial \,\text{ vec}\,{\varvec{P}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right)^{\prime }\otimes \varvec{1}_{k^2}\right) \nonumber \\&\times \left(\varvec{1}_m\otimes \varvec{K}^{(m,k)}\otimes \varvec{1}_k\right)\Big (\varvec{1}_{km}\otimes \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}+ \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}\otimes \varvec{1}_{km}\Big ) \frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }},\nonumber \\ \end{aligned}$$

(114)

with the expressions for the derivatives and second derivatives of ${\varvec{G}}_{\varvec{\theta }}$ given in (95) and (111). Further

$$\begin{aligned} \frac{\partial ^2{\varvec{\xi }}_{\varvec{\theta }}}{\partial {\varvec{\theta }}\partial \varvec{\theta }^{\prime }}&= \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }^{\prime }}({\varvec{v}}_{\varvec{\theta }}^{\prime }\otimes \varvec{1}_k)\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} + \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }^{\prime }}{\varvec{G}}_{\varvec{\theta }}\frac{\partial {\varvec{v}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\nonumber \\&= \left(\left(\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\right)^{\prime }\otimes \varvec{1}_k\right) \Big (\varvec{1}_m\otimes \,\text{ vec}\,\varvec{1}_k\Big )\frac{\partial \,\text{ vec}\,{\varvec{v}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\nonumber \\&+ (\varvec{1}_p\otimes ({\varvec{v}}_{\varvec{\theta }}^{\prime }\otimes \varvec{1}_k)) \frac{\partial ^2\,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }\partial \varvec{\theta }^{\prime }} \nonumber \\&+ \left(\frac{\partial \,\text{ vec}\,{\varvec{G}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \right)^{\prime }\frac{\partial {\varvec{v}}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}, \end{aligned}$$

(115)

and

$$\begin{aligned} \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }^{\prime }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1} = -({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\otimes {\varvec{\varOmega }}_{\varvec{\theta }}^{-1})\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}, \end{aligned}$$

(116)

and

$$\begin{aligned} \,\text{ d}\!\,\left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right)&= -{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}(\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}){\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}+ {\varvec{\varOmega }}_{\varvec{\theta }}^{-1}(\,\text{ d}\!\,{\varvec{u}}_{\varvec{\theta }}){\varvec{\mu }}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\end{aligned}$$

(117)

$$\begin{aligned}&+{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}(\,\text{ d}\!\,{\varvec{u}}_{\varvec{\theta }}^{\prime }){\varvec{\varOmega }}_{\varvec{\theta }}^{-1} - {\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}(\,\text{ d}\!\,{\varvec{\varOmega }}_{\varvec{\theta }}){\varvec{\varOmega }}_{\varvec{\theta }}^{-1}.\qquad \end{aligned}$$

(118)

By using the identity $\,\text{ vec}\,(ABC) = (C^{\prime }\otimes A)\,\text{ vec}\,(B)$ several times we obtain

$$\begin{aligned} \frac{\partial \,\text{ vec}\,}{\partial \varvec{\theta }^{\prime }}{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }} {\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}&= -\left(\left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right]^{\prime }\otimes {\varvec{\varOmega }}_{\varvec{\theta }}^{-1} \right)\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}\end{aligned}$$

(119)

$$\begin{aligned}&- \left(\left[{\varvec{u}}_{\varvec{\theta }}^{\prime }{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right]^{\prime }\otimes {\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right) \frac{\partial \,\text{ vec}\,{\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \end{aligned}$$

(120)

$$\begin{aligned}&- \left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\otimes \left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}\right]\right) \frac{\partial \,\text{ vec}\,{\varvec{\xi }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }} \end{aligned}$$

(121)

$$\begin{aligned}&- \left({\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\otimes \left[{\varvec{\varOmega }}_{\varvec{\theta }}^{-1}{\varvec{u}}_{\varvec{\theta }}{\varvec{u}}_{\varvec{\theta }}^{\prime } {\varvec{\varOmega }}_{\varvec{\theta }}^{-1}\right]\right)\frac{\partial \,\text{ vec}\,{\varvec{\varOmega }}_{\varvec{\theta }}}{\partial \varvec{\theta }^{\prime }}, \end{aligned}$$

(122)

and the second order derivative of the log-likelihood (107) now follows from applying the product rule with (114), (115), (116) and (119).