Non-Negative Variance Component Estimation for Mixed- Effects Models Volume 2 - Issue 3

Jaesung Choi*

• Department of Statistics, Keimyung University, Korea

Received: February 27, 2020;   Published: March 11, 2020

*Corresponding author: Department of Statistics, Keimyung University, Korea

DOI: 10.32474/CTBB.2020.02.000140

Abstract PDF

Abstract

This paper discusses projection methods to find nonnegative estimates for variance components of random effects in mixed models. The proposed methods are based on the concepts of projections, which are called projection method I, II and III. These three methods produce the same nonnegative estimates for the same data. Even though each method uses orthogonal projections in its own way, the results are the same for the variance components regardless of which method is used. It is shown that quadratic forms of an observation vector are constructed differently in each method. All sums of squares in quadratic forms of the observation vector can be expressed as squared distances of corresponding projections. A projection model is defined and used to evaluate expected values of quadratic forms of observations that are associated with variance components. Hartley’s synthesis is used as a method for finding the coefficients of variance components.

Keywords:Mixed model; projections; quadratic forms; random effects; synthesis

Introduction

Much literature has been devoted to the estimation of variance components in random effects or mixed effects models. A variance component should always be nonnegative by its definition; however, we sometimes get it as negative [1,2]. illustrated this with the simple hypothetical data of a one-way classification having three observations in two classes and insisted that there was nothing intrinsic in the analysis of variance method to prevent it. When a negative estimate happens, it is not easy to handle this situation in interpretation and action. Hence, many papers have been contributed to strategies to deal with the negative values as estimates of variance components [3]. suggests that negative estimates of variance components can occur in certain designs such as split plot and randomized block designs by random- inaction. Thompson discusses the interpretation of the negative estimate and suggests an alternative method when the analysis of variance method yields negative estimates [4]. also suggest a procedure for eliminating negative estimates of variance components in random effects models. The analysis of the variance method is almost exclusively applied to balanced data for estimating variance components. However, there are multiple methods for unbalanced data. Therefore, it is necessary to identify the types of data before choosing a method. Though balanced data have the same numbers of observations in each cell, unbalanced data have unequal numbers of observations in the subclasses made by the levels of classification factors. Depending on the types of data, many methods can be applied to the estimation of variance components in a vector space. Representing data as vectors, the vector space of an observation vector can be partitioned in many ways, depending on the data structure. For balanced data, the vector space can always be partitioned into orthogonal vector subspaces according to the sources of variation, but it is not true for unbalanced data. This is the main difference between balanced and unbalanced data from the view point of a vector space. A random effect is a random variable representing the effect of a randomly chosen level from a population of levels that a random factor can assume, while a fixed effect is an unknown constant denoting the effect of a predetermined level of a factor. A linear model with these two types of effects is called a mixed effects model. The primary concern with the model in this paper is naturally in the nonnegative estimation of variance components of random effects. A negative estimate can happen in any method that contributes to the estimation.
Hence, many papers have investigated strategies for interpretation and alternatives. Such strategies are seen in [5-9]. However, it is necessary to have a method that yields nonnegative estimates despite all such efforts [10].suggested a method that uses reductions in sums of squares due to fitting both the full model and different sub-models of it for estimating variance components of random effects in mixed models. This method is called the fitting constants method or Henderson’s Method 3. Even though it has been used extensively for the estimation of variance components in mixed models, it still has some defects producing negative estimates in case [11]. synthesis is also used for calculating the coefficients of variance components in the method. Although this method is very useful, we should recognize whether quadratic forms for variance components are in the right form or not. Otherwise, expectations of the quadratic forms can be different from the real ones. This is going to be discussed in detail in projection model building. This paper suggests three methods to produce nonnegative estimates for variance components in mixed models. They are based on the concept of projection defined on a vector space. The definition of a projection and its related concepts are discussed in [12,13]. Quadratic forms in the observations can be obtained as squared distances of projections defined in proper vector subspaces. Each method requires that all vector subspaces for projections should be orthogonal to each other at the stage of fitting sub-models serially. When the orthogonality is satisfied with vector subspaces, it is possible to get nonnegative estimates. Hence, we also discuss how to construct orthogonal vector subspaces from a given mixed model. Quadratic forms as sums of squares due to random effects are then used to evaluate their expected values. Hereafter, equating quadratic forms to their expected values represents available equations for the estimates. For calculating the coefficients of variance components, Hartley’s synthesis is applied but in a different manner, which will be discussed.

Mixed Models

Mixed models are used to describe data from experimental situations where some factors are fixed, and others are random. When two types of factors are considered in experiments, one is interested in both parts, that is, the fixed-effects part and the random-effects part, in models. Let α be a vector of all the fixed effects except μ in a mixed model and let δ_i denote a set of random effects for random factor i for i = 1, 2, r. Then, i δ could be interaction effects or nested-factor effects when they are simply regarded as effects from random factors. The matrix notation of the mixed model for an observation vector y is

where jμ + X_Fα_F is the fixed part of the model and X_Rδ_R + ∈ is the random part of the model. δ_i s are assumed to be independent and identically distributed as N(0, δ_σi² I ), and ε is assumed to be distributed as N(0,σ_∈²I ) . The mean and variance of y from (1) is

The expectation of any quadratic form in the observations of a vector y is represented as a function of variance components and fixed effects. The variance components of the full model can be estimated by the fitting constants method of using reductions in the sums of squares due to fitting the full model and the submodel of it. This method provides unbiased estimators of the variance components that do not depend on any fixed effects in the model, and it has been widely used for the estimation of variance components for unbalanced data. However, it still has an unsolved problem having negative solutions as estimates. As an alternative, a method which is based on the concepts of projections is suggested [14]. To discuss it, we consider the model (1) as representative. Since there are two parts in the model, we naturally divide the model into a fixed part and a random part. The random part of the model consists of random effects and errors:

where ∈_R = Σ_i=1^r X_iδ_i+ ∈ The general mean μ and fixed effects α_F of (5) can be estimated from normal equations. Regarding y as an observation vector in the n-dimensional vector space, it can be decomposed into two component vectors orthogonal to each other. The decomposition of y is done by projecting y onto the vector subspace generated by ( j, X_F ) .

Projection method

Since a method based on the concept of projection is discussed, it will be called the projection method. For a mixed model such as (5), we can decompose y into two components by means of projections. Denoting (j, X_F) and (μ, α_F)^T by X^y_m and μ_m , respectively, the projection of y onto the vector subspace spanned by X_m is X_m X _my − , where X_m − denotes a Moore-Penrose generalized inverse of XM . Then, y can be decomposed into two vectors, that is, X_m X −_my and (I -X_MX−_M )y − which are orthogonal [15,16]. Instead of the fitting constants method, the projection method is attempted to estimate the nonnegative estimates of the variance components in a mixed model. To explain the method simply, suppose there are two factors A and B for a two-way crossclassified unbalanced data where A is fixed with a levels and B is random with b levels. The model for this is

where y is an observation vector in the n dimensional vector space, α_F is a vector of fixed effects of A, δ_β and δ_αβ represent vectors of random effects of B and AB interaction respectively, and X_M = ( j, X_F ), α_M(μ, α_F)^T and ∈_M = X_βδ_β + X_αβ δ_αβ +∈. The second εexpression of (6) represents the fixed-effects part and the random part. The random part S_M is obtained by the projection of y onto a vector subspace generated by the X_M, which is (I -X_M X_M−) y. So, y is represented as

where y_M= X_M X_M−y satisfies the two conditions for being the projection of y onto a vector subspace spanned by the columns of X_M. The projection should be obtained by the orthogonal projection to the subspace and denoted as a linear combination of the column vectors of X_M.X_M X_M-y conditions. Since y_M is orthogonal to e_M, the random part e_M=(I− X_M X_M-)y is not affected by the fixed effects and has all the information about the variance components and random error variance. Since there are two random effects and random error terms in the model of (6), we can use e_M for finding the related variance components. The model for the estimation of σ²_β is

where y_B X_BX_B−e_M = is the projection of e_M onto the column space of X_B ⋅ y_B and e_B are orthogonal each other. Hence, e_B is not affected by the random effects B δ Therefore, e_B is used for finding the subspace that has information about σ²_αβ. The model for this is

where X_AB =(I − X_M X_M X⁻_BX⁻_B) X_αβ and ∈_αβ=(I− X_M X⁻_M X_B X⁻_B) Hence, the projection of eB onto the subspace generated by X_AB is y_AB= X_AB X⁻_ABe_B. Then,

where e_AB is (I − X_AB X⁻_AB )e_B. Finally, we can use e^AB for finding the coefficient matrix of the random error vector which generates the error space orthogonal to all the other spaces.

Thus, we can know that e_AB has all the information about σ²_∈ of the random error vector ∈ . Denoting y as the sum of orthogonal projections and error part,

Each term of (13) can be used to calculate the sums of squares that are quadratic forms in the observations. Since y is partitioned as four terms, there are four available sums of squares. We denote them SS_M, SS_B, SS_AB and SS_E where subscripts are corresponding factors. They are defined as

where each SS term is given as the squared length of the projection of y onto its own vector subspace, and X_E= (I−X_MX⁻_M X_BX⁻_B X_AB X⁻_AB). All the sums of squares are evaluated by using the eigenvalues and eigenvectors of the projection matrices associated with the quadratic forms in y. Since projections are defined on subspaces that are orthogonal to each other, we can identify the coefficient matrices spanning them.

Projection model

Since y is made up by the sum of mutual orthogonal projections such as (13), y can be represented by the orthogonal coefficient’s matrices of the effects of the assumed model (6). Temporarily, we denote y as y_p for differentiating the model based on projections to the classical model (6). Then, the model for y_p is

y = X_Mα_M + X_Bα_β + X_ABδ_αβ + X_E ∈ (15)

where y_p = y . Since each coefficient matrix of the effects is derived from the corresponding orthogonal projection, the equation of (15) defines a projection model that is different from a classical two-way linear mixed model (6). It is useful for evaluating the coefficients of the variance components in the expectations of the quadratic form of an observation vector y_p . In the model, all the coefficient matrices are orthogonal to

each other. δ_β, δ_αβ and s are assumed to be N(0,σ²_βI_b), N(0,σ²_αβI_ab) and N(0,σ²_∈I_n) respectively. The expectation and the covariance matrix of y p of the projection model (15) is

Equating the three sums of squares, SS_B, SS_AB, and SS_E of (14) to their correspond- ing expectations leads to linear equations in the variance components, the solutions to which are taken as the estimators of those components. Now, the equations are

Solutions from the linear equations (18) are nonnegative estimates of the variance components. Since there are three different ways of getting sums of squares by means of projections, we will differentiate them with projection method I, II, and III. The procedure using the system of linear equations like (18) is called projection method I. The projection method II uses residual vectors after projecting y onto orthogonal subspaces. That is, e_M, e_B, and e_AB are used such as. Then

Since e_M has three random components, e_M^T e_M in the quadratic form of y_p in which the coefficients matrices of the projection model are orthogonal is available for esti- mating their variance components. Denoting e_M^T e_M as RSS_M,

RSS_M = e_M^T e_M, (20)

where M RSS measures the variation due to the three random effects, and thus, the quantity is used for the estimation of three variance components σ²_β , σ²_αβ, and σ²_∈. Representing the residual random vector e_B as y_p , has two random components as follows.

Hence, e^T_B is used as an variation quantity for two random effects vectors. Denoting e^T_B as RSS_B,

where RSS_B is used for estimating the two variance components σ_2αβ and σ_2∈ since e_B has just two random effects. Finally, expressing e_AB as y_p

e_AB = (I − X_AB X_AB⁻) e_B = X_E ∈ (23)

which has just one random component s. Therefore, e_AB^T e_AB shows the variation due to the random error vector only, and this quantity is used for estimating the variance component σ² ∈ . Denoting e_AB^T e_AB as RSS_AB,

RSS_AB = e_AB^T e_AB (24)

Hence,RSS_M , RSS_B , and RSS_AB are another set of sums of squares for estimating variance components instead of using sums of squares derived from the projections as an alternative method.RSS_M , RSS_B , and RSS_AB are also evaluated by using the eigenvalues and eigenvectors of the projection matrices associated with the quadratic forms in y. Now, the expected values of the RSS’s are

Then, the linear equations of variance components are obtained by equating the RSS’s to their expected values, the solutions for which always produce nonlinear estimates.
That is,

Even though two systems of linear equations are not the same, either system will produce the same estimates of the variance components that are nonnegative. As another method, projection method III is also available for the estimation of variance components. This method is done as follows. For the model of (6), y = Xθ +∈, where X = (j, X_F, X_β, X_αβ) and θ =( μ ,α_F, δ_β, δ_αβ )^T. This method splits the vector space of an observation vector into two subspaces, one for the projection part and the other for the error part at each step. Then, the projection of y onto the subspace spanned by XX⁻ is given by XX⁻ y , and the error vector in the error vector space is (I − XX⁻ ) y . Therefore, the coefficient matrix of s is derived as (I − XX⁻ )from it. The quadratic form y '(I − XX⁻ ) y denoted by 0 BSS is the sum of squares due to random error only, which has all the information about σ²_∈ . For information about both σ²_αβ and σ²_∈, the vector space of the observation vector can be decomposed into two parts one for the projection part and the other for the error part. For this, the model to be fitted is y = X₁ θ₁ +∈₁ , where X₁ (j j, X_F, X_β ) θ₁ = (μ, α_F, δ_β ) and ∈₁= X_αβ α_αβ +∈. Then, the projection of y onto the subspace spanned by X₁ X₁⁻ is given by X₁ X₁⁻ y, and the error vector in the errror vector space is (I-X₁ X₁⁻)y . The quadratic form y^T(I-X₁ X₁⁻)y denoted by BSS₁ has information about σ²_αβ and σ²_∈ . Now, the error vector is represented by

Hence, the coefficient matrix of δ_β is given by (I-X₂X₂⁻ )X_β. It is necessary to evaluate the expected values of the quadratic forms for constructing the equations for the variance components. They are

The nonnegative estimates of variance components are given as solutions of linear equations of σ²_α and σ²_αβ σ²_∈. The above equations are summarized as follows:

where c_ij ’s are coefficients of variance components of expected values of quadratic forms of (29).

Examples

As a first example of nonnegative estimates of random effects for a two-way mixed model, Montgomery (2013)’s data are illustrated. The data are from an experiment for a gauge capability study where parts are randomly selected, and three operators are fixed. An instrument or gauge is used to measure a critical dimension on a part. Twenty parts have been selected from the production process, and only three operators are assumed to use the gauge. The assumed model for the data in Table 1 is y_ijk = μ + α_i +γ_j + (αγ)_ij + ∈_ijk, where they α_i (i = 1, 2, 3) are fixed effects such that Σ_i=1³α_i = 0 and γ_j (j = 1, 2,...., 20), (αγ)_ij, and ∈ijk are uncorrelated random variables having zero means and variances V_ar(γj)= σ_γ², V_ar((αγ)_ij)= σ_αγ² V_ar(∈_ijk)= σ². Under the assumed unrestricted model, estimated variance components are σ^2ˆ_{γ = 10.2798 , σ^2ˆαγ= −0.1399 , and σ^2ˆ∈ = 0.9917 . Applying the projection method, I to the data, the linear equations of variance components are given as follows:}

The solutions of the equations are σ^2ˆ_{γ = 10.3985, σ^2ˆαγ= 0.3559, σ^2ˆ∈ = 0.9917. All the variance components are estimated nonnegatively. When we apply projection method II to the same data, we get}

where RSS_fixed = 1271.975 , RSS_part = 86.55 , and RSS_{part×operator} = 59.5 . The solutions for the equations are σˆ²_γ =10.3985, σˆ²_αγ =0.3559, σˆ²_∈ =0.9917 which are the same as the previous solutions. Hence, either one of the projection methods can be used for the nonnegative estimation of variance components of random effects in a mixed model. Projection method III also gives the same result as projection methods I and II for the data. As a second example, Searle [2]’s hypothetical data are illustrated. Searle explains why a negative estimate can occur in the estimation of variance component of random effects in a random model. The data are shown in Table 1. Since class in Table 2 is a random factor, the one-way random effects model is assumed. The assumed model is y_ij = μ+α_i+ ∈_ij, where the α_i(i=1,2) are random effects and ∈_ij are uncorrelated random errors having zero means and variances V_ar(α_i)= σ²_α and V_ar(∈_ij)= σ². As a result of the analysis of variance, the estimates of variance components are given as σˆα2 = −15.33 and σˆ∈2 = 52 . Searledemonstrated how negative estimates could come from the analysis of variance and insisted that there would be nothing intrinsic in the method to prevent it. However, the projection methods yield the same nonnegative estimates as σˆ²_α = 2 and σˆ²_∈ = 52 in any method.

Conclusion

Variance should be a nonnegative quantity as a measure of variation in data by its definition. In this work, it shows that orthogonal projections are very useful for defining a projection model for nonnegative variance estimation. Although there have been many attempts in literature to fix the problem of negative estimates for variance components over decades, those were not successful. However, the proposed methods in this paper always produce nonnegative estimates of variance components of the random effects in a mixed model. The two most important findings are checked and discussed for the estimation of nonnegative variance component. One is that a projection model should be derived from an assumed mixed-effects model. The other is that expectations of quadratic forms associated with the random effects should be evaluated from the projection model. This paper introduces terms such as projection method I, II, and III related to the methods, and the projection model for emphasizing projection rather than model fitting. Though they are based on the same assumed model, three methods are ap- plied differently in the application. Each method uses in its own way but summing up all orthogonal projections come to the observation vector. Depending on the types of projections, each method produces three different sets of equations for the evaluation of quadratic forms. Nonetheless, all of them show the same nonnegative estimates for variance components. It also shows that projection methods can be used for estimating variance components of the random effects in either random model or mixed model through examples. It should be noted that all the matrices associated with the quadratic forms come from the projection model not from the assumed model. In such a case, Hartley’s synthesis can yield correct coefficients of variance components.

Funding

This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education under Grant No.2018R1D1A1B07043021.

References

1. Searle SR (1971) Linear models. John Wiley and Sons New York, USA p. 1.
2. Searle SR, Gruber MHJ (2016) Linear Models. John Wiley and Sons New York, USA.
3. Nelder J (1954) A The interpretation of negative components of variance. Biometric 41: 544-548.
4. Milliken GA, Johnson DE (1984) Analysis of messy data volume 1: designed experiments. Van Nostrand Reinhold New York, USA.
5. Searle SR, Fawcett RF (1970) Expected mean squares in variance components models having finite populations. Biometrics 26(2): 243-254.
6. Hill BM (1965) Inference about variance components in the one-way mode. J Am Stat Assoc 60(311): 806-825.
7. Hill BM (1967) Correlated errors in the random model. J Am Stat Assoc 62(320): 1387-1400.
8. Searle SR, Casella G, McCulloch CE (2009) Variance components. John Wiley and Sons New York, USA.
9. Harville DA (1969) Variance component estimation for the unbalanced one-way random classification a critique. Aerospace Research Laboratories ARL p. 69.
10. Henderson CR (1953) Estimation of variance and covariance components. Biometrics 9(2): 226-252.
11. Hartley HO (1967) Expectations variances and covariances of ANOVA means squares bysynthesis. Biometrics 23: 105-114.
12. GA (1983) Franklin Matrices with Applications in Statistics. Wadsworth Inc New York, USA.
13. Johnson RA, Wichern DW (2014) Applied multivariate statistical analysis. Prentice hall Upper Saddle River New Jersey, USA.
14. Montgomery DC (2013) Design and analysis of experiments. John Wiley and Sons New York, USA.
15. Thompson WA (1967) Negative estimates of variance components: an introduction Bulletin International Institute of Statistics 34: 1-4.
16. Thompson WA, Moore JR (1967) Non negative estimates of variance components. Tech no metrics 5(4): 441-449.