This contribution is devoted to the estimation of the parameters of multivariate Gaussian mixture where the covariance matrices are constrained to have a linear structure such as Toeplitz, Hankel, or circular constraints. We propose a simple modification of the expectation-maximization (EM) algorithm to take into account the structure constraints. The basic modification consists of virtually updating the observed covariance matrices in a first stage. Then, in a second stage, the estimated covariances undergo the reversed updating. The proposed algorithm is called the inverse EM algorithm. The increasing property of the likelihood through the algorithm iterations is proved. The strict increasing for nonstationary points is proved as well. Numerical results are shown to corroborate the effectiveness of the proposed algorithm for the joint unsupervised classification and spectral estimation of stationary autoregressive time series.