Agricultural Statistics & Data Science · Statistical theory core · also Regression Analysis, Linear Models, Applied Regression Analysis
The general linear model in matrix form: estimability, the Gauss-Markov theorem, least squares with full and less-than-full rank, the analysis of variance as a linear model, model checking and selection, and regression diagnostics. It is the theory under every design in Phase 3.
In the exams: AICE-JRF Ag Stat Units 2 and 4 (heavy); GATE ST 'Regression Analysis' (heavy); UPSC ISS Statistics II (heavy); GATE DA regression (present)
The linear model is the common engine of regression, the analysis of variance and the analysis of covariance. This subject presents it in matrix form, as master's and PhD courses in agricultural statistics do: least squares and the normal equations, estimability and the Gauss-Markov theorem, models of less than full rank and generalised inverses, the distribution of quadratic forms and Cochran's theorem, tests of general linear hypotheses, fixed, random and mixed effects with nested and crossed classifications, multiple and partial correlation, residual analysis, leverage and influence, multicollinearity and ridge regression, model selection, weighted and generalised least squares, and non-linear regression.
The designs of the basic designs and factorial subjects of this track are all linear models; this subject explains why their ANOVA tables work, why degrees of freedom must add up, and how to handle what designed experiments do not: unbalanced data, continuous predictors, unequal variances and correlated errors. Econometric extensions (simultaneous equations, panel data, instrumental variables) are taught in the Agribusiness econometrics subject and in this track's causal inference subject, and are not repeated here.
In examinations, ICAR AICE-JRF/SRF Agricultural Statistics names linear models (random, fixed and mixed effects, nested and crossed classifications, the Gauss-Markov theorem) in Unit 4 and correlation and regression with model adequacy and selection of the best regression in Unit 2. GATE Statistics has a regression section covering simple and multiple regression, R squared and adjusted R squared, quadratic forms, the Fisher-Cochran and Gauss-Markov theorems and tests and intervals for coefficients; GATE Data Science covers regression and ridge regression; and the Indian Statistical Service papers include linear models and regression (as reported).
The general linear model writes n observations as y = X beta + e, where y is the n x 1 vector of responses, X is an n x p matrix of known constants (the design or model matrix), beta is a p x 1 vector of unknown parameters and e is a vector of random errors. The standard assumptions are E(e) = 0 and V(e) = sigma squared I: errors have mean zero, a common variance and are uncorrelated. For tests and confidence intervals the errors are further assumed normal. The word linear refers to the parameters, not to the explanatory variables: y = b0 + b1 x + b2 x squared + e is a linear model, because it is linear in b0, b1 and b2, while y = a exp(bx) + e is not. Regression with continuous predictors, the analysis of variance for designed experiments (where X contains zeros and ones indicating treatments and blocks) and the analysis of covariance are all special cases. That unity is the reason this subject underlies every design in the basic designs and factorial subjects of this track.
The least squares estimate of beta minimises the residual sum of squares (y - X beta)'(y - X beta). Setting the derivative to zero gives the normal equations X'X beta-hat = X'y. When X has full column rank p, X'X is non-singular and beta-hat = (X'X)^(-1) X'y. Its expectation is beta (it is unbiased) and its variance matrix is sigma squared (X'X)^(-1). The fitted values are y-hat = X beta-hat = H y, where H = X (X'X)^(-1) X' is the hat matrix, symmetric and idempotent (HH = H), with trace equal to p. The residuals are e-hat = (I - H) y, and the residual sum of squares is RSS = y'(I - H) y. Because E(RSS) = (n - p) sigma squared, the unbiased estimator of sigma squared is s squared = RSS / (n - p), the residual mean square, with n - p degrees of freedom.
With one predictor, X has a column of ones and a column of x values. The normal equations give the slope b = Sxy / Sxx and intercept a = y-bar - b x-bar, where Sxx is the corrected sum of squares of x and Sxy the corrected sum of products. The regression sum of squares is b Sxy, the residual sum of squares is Syy - b Sxy, and the variance of b is sigma squared / Sxx, so the standard error of b is the square root of (s squared / Sxx) and t = b / SE(b) on n - 2 degrees of freedom tests the slope. The variance of a fitted mean at x0 is sigma squared [1/n + (x0 - x-bar) squared / Sxx], smallest at the mean of x, which is why predictions far from the data are imprecise.
In a nitrogen trial with n = 12 plots, suppose the corrected sums are Sxx = 40, Sxy = 60 and Syy = 120, with x-bar = 5 and y-bar = 20. The slope is b = 60/40 = 1.5 and the intercept a = 20 - 1.5 x 5 = 12.5. The regression sum of squares is b Sxy = 90, the residual sum of squares 120 - 90 = 30 on 10 degrees of freedom, so s squared = 3. R squared = 90/120 = 0.75. The standard error of the slope is the square root of (3/40) = 0.274, and t = 1.5/0.274 = 5.48 on 10 degrees of freedom; equivalently F = 90/3 = 30 = t squared. The example shows how the scalar formulas of a first course are the matrix formulas written out for two parameters.
A linear function lambda' beta of the parameters is estimable if there exists a linear function a'y of the observations whose expectation is lambda' beta for every beta. Since E(a'y) = a'X beta, lambda' beta is estimable if and only if lambda' = a'X for some a, that is, lambda' lies in the row space of X. When X has full column rank, every linear function of beta is estimable. In models of less than full rank, as in the one-way ANOVA written with a general mean and t treatment effects, individual parameters such as mu or tau_1 are not estimable, but treatment means mu + tau_i and contrasts among the tau_i (with coefficients summing to zero) are. An equivalent test, useful in practice, is that lambda' beta is estimable if and only if lambda' G X'X = lambda' for a generalised inverse G of X'X. The estimable functions are exactly the quantities that the data can tell us about, whatever constraints are imposed to solve the normal equations.
Under E(e) = 0 and V(e) = sigma squared I, with no normality assumption, the Gauss-Markov theorem states that for any estimable function lambda' beta, the estimator lambda' beta-hat, where beta-hat is any solution of the normal equations, is the best linear unbiased estimator (BLUE): it has the smallest variance among all linear unbiased estimators. In full-rank regression this means each least squares coefficient is BLUE. The theorem needs uncorrelated errors with equal variance. When the variances differ or the errors are correlated, ordinary least squares is still unbiased but no longer best; generalised least squares (chapter 12) is then BLUE. If the errors are also normal, least squares estimates coincide with maximum likelihood estimates and are minimum variance among all unbiased estimators, linear or not.
In the one-way classification y_ij = mu + tau_i + e_ij with t treatments, the model matrix has a column of ones for mu and t indicator columns for the treatments. The indicator columns add up to the column of ones, so X has rank t, not t + 1, and X'X is singular: the normal equations have infinitely many solutions. The same happens in block designs, factorial models with all main effects and interactions, and any model with more parameters than the data can separate. Two approaches deal with this. One imposes constraints, such as the sum of the tau_i = 0 or tau_t = 0, which pick a particular solution; software uses such constraints by default. The other uses a generalised inverse. In both cases the estimates of individual parameters depend on the choice, but the estimates of estimable functions, the fitted values, the residuals, the residual sum of squares and the tests do not.
A generalised inverse (g-inverse) of a matrix A is any matrix G such that A G A = A. For a symmetric matrix such as X'X, a g-inverse can be found by deleting rows and columns to leave a non-singular submatrix of full rank, inverting it and putting zeros back. Then beta-hat = G X'y solves the normal equations, and for an estimable lambda' beta the estimate lambda' beta-hat is the same whichever G is used. The Moore-Penrose inverse is the unique g-inverse satisfying four conditions: AGA = A, GAG = G, AG symmetric and GA symmetric. It gives the minimum-norm least squares solution. In a model of rank r(X) less than p, the residual degrees of freedom are n - r(X), not n - p: in the one-way classification with N observations and t treatments, error df = N - t.
Take a one-way layout with three treatments and two observations each. The model matrix has four columns (mu and three treatment indicators) and rank 3. Imposing the constraint that the treatment effects sum to zero gives mu-hat equal to the grand mean and tau_i-hat equal to the treatment mean minus the grand mean; imposing tau_3 = 0 instead gives mu-hat equal to the mean of treatment 3 and tau_i-hat equal to the difference between treatment i's mean and treatment 3's. The individual estimates differ, but every treatment mean mu + tau_i, every contrast such as tau_1 - tau_2, the fitted values and the residual sum of squares (on 6 - 3 = 3 degrees of freedom) are identical under both choices, as the theory says they must be.
Sums of squares are quadratic forms y'Ay. Their distributions under normality justify the F tests of regression and ANOVA. If y is distributed as N(mu, sigma squared I) and A is symmetric, then y'Ay / sigma squared has a (non-central) chi-square distribution if and only if A is idempotent; its degrees of freedom are the rank of A (equal to its trace for an idempotent matrix) and its non-centrality parameter is mu'A mu / sigma squared (some texts define it with a factor 1/2). When mu'A mu = 0, the distribution is central. Two quadratic forms y'Ay and y'By are independent if and only if AB = 0, and a quadratic form y'Ay and a linear form By are independent if and only if BA = 0. For example, the sample mean and the corrected sum of squares are independent, because the corrected sum of squares uses A = I - J/n, where J is the matrix of ones, and the mean uses the vector of 1/n, whose product with A is zero; this matrix has rank n - 1, so the corrected sum of squares over sigma squared is chi-square with n - 1 degrees of freedom.
Cochran's theorem (also called the Fisher-Cochran theorem) states that if y'y = sum of y'A_i y, where the A_i are symmetric matrices with ranks r_i and y is N(0, sigma squared I), then the quadratic forms y'A_i y / sigma squared are independent chi-square variables with r_i degrees of freedom if and only if the sum of the ranks equals n. It underlies the ANOVA table: the total sum of squares splits into treatment, block and error sums of squares whose degrees of freedom add up, so each is an independent chi-square (central under the null hypothesis) and their ratios, divided by degrees of freedom, follow F distributions. This is the formal reason why degrees of freedom must add up in an ANOVA, and why a mistake in partitioning, such as counting the same contrast in two lines, destroys the validity of the tests.
In full-rank regression with normal errors, RSS / sigma squared = y'(I - H)y / sigma squared is central chi-square with n - p degrees of freedom, because I - H is idempotent with trace n - p and (I - H)X beta = 0. The regression sum of squares about the mean uses H - J/n, also idempotent, with rank p - 1, and (H - J/n)(I - H) = 0, so the two sums of squares are independent and their mean-square ratio is F with p - 1 and n - p degrees of freedom; under the hypothesis that all slopes are zero the regression form is central. This short argument is what GATE and ISS examiners expect when they ask why the overall F test is valid.
A general linear hypothesis H0: L beta = c, where L is a q x p matrix of rank q (each row an estimable function), covers tests of single coefficients, of several coefficients at once, of equality of coefficients and of all treatment contrasts. The test statistic is F = (L beta-hat - c)'[L (X'X)^(-1) L']^(-1)(L beta-hat - c) / (q s squared), which under H0 and normal errors follows F with q and n - p degrees of freedom. An equivalent and often simpler route is the extra sum of squares principle: fit the full model and the reduced model that satisfies H0, and compute F = [(RSS_reduced - RSS_full) / q] / [RSS_full / (n - p)]. The analysis of variance for any design is a sequence of such comparisons. In regression, the overall F test for all slopes being zero compares the full model with the model containing only the intercept.
The coefficient of determination R squared = SSR / SST = 1 - RSS / SST, where SST is the corrected total sum of squares, is the proportion of variation explained by the regression. For simple linear regression it is the square of the correlation coefficient. R squared never decreases when a predictor is added, even a useless one, so it cannot be used alone to compare models of different sizes. The adjusted R squared corrects for the number of predictors k (excluding the intercept): adjusted R squared = 1 - (1 - R squared)(n - 1) / (n - k - 1). It can fall when a weak predictor is added and can be negative. The overall F statistic can be written from R squared: F = [R squared / k] / [(1 - R squared) / (n - k - 1)]. A high R squared does not prove that the model is correct, that the relation is causal, or that predictions will be accurate outside the range of the data.
A confidence interval for a single coefficient is beta-hat_j plus or minus t (n - p) times its standard error, the square root of s squared times the j-th diagonal element of (X'X)^(-1). The confidence interval for the mean response at x0 uses the variance s squared x0'(X'X)^(-1) x0, while the prediction interval for a new observation adds s squared for the new error, so prediction intervals are always wider than confidence intervals for the mean. Joint confidence regions for several coefficients are ellipsoids; simultaneous intervals use Bonferroni or Scheffe adjustments.
A regression of yield on k = 3 weather variables from n = 30 seasons gives R squared = 0.60. The overall F statistic is (0.60/3) / (0.40/26) = 0.20 / 0.01538 = 13.0 on 3 and 26 degrees of freedom, far beyond the 5% critical value of about 2.98, so the predictors jointly explain significant variation. The adjusted R squared is 1 - 0.40 x 29/26 = 0.554. With 25 observations, k = 4 and R squared = 0.80, the adjusted value would be 1 - 0.20 x 24/20 = 0.76.
An effect is fixed when its levels are chosen deliberately and conclusions are confined to those levels, such as specific fertiliser rates or released varieties. It is random when the levels are a random sample from a larger population and interest lies in the variation of that population, such as sires sampled from a breed, locations sampled from a region or batches of a product. Models with only fixed effects (apart from error) are fixed models, those with only random effects random models, and those with both mixed models. In a one-way random model y_ij = mu + a_i + e_ij, with a_i having variance sigma_a squared, the expected mean squares are E(MSA) = sigma squared + n sigma_a squared and E(MSE) = sigma squared, with n observations per group. The ANOVA estimator of the variance component is sigma_a squared-hat = (MSA - MSE) / n, which can be negative when MSA < MSE; the intraclass correlation sigma_a squared / (sigma_a squared + sigma squared) measures how alike observations in the same group are, and in animal breeding it underlies heritability and repeatability estimates. REML estimation of variance components is treated in the mixed-models subject.
Two factors are crossed when every level of one occurs with every level of the other, as in a factorial experiment; their interaction can then be estimated. A factor B is nested within A when each level of B occurs with only one level of A, as with fields within farms, plants within plots or sires within breeds; there is then no A x B interaction, and the B variation is estimated within each level of A. In a two-stage nested design with a levels of A, b levels of B within each and n observations per B, the degrees of freedom are a - 1 for A, a(b - 1) for B within A and ab(n - 1) for error. Which mean square is the correct denominator for an F test follows from the expected mean squares. In a nested design with random B, A is tested against the B-within-A mean square, not against the error, because the variation among levels of A includes the variation among the B units inside them. In a two-factor crossed mixed model with A fixed and B random, A is tested against the A x B interaction. Getting the denominator wrong is the most common mistake in analyses of designs with random effects, and it is the same issue as the separate error terms of split plots.
Suppose 3 farms are sampled, 4 fields within each farm and 2 soil cores per field. The degrees of freedom are 2 for farms, 3 x 3 = 9 for fields within farms, 12 x 1 = 12 for cores within fields, 23 in total. If fields are random, the test of farms uses the fields-within-farms mean square as denominator (2 and 9 degrees of freedom), not the core-to-core mean square (12 degrees of freedom), because farm means vary partly because they contain different fields. For a one-way random model with MSA = 50, MSE = 10 and n = 5 per group, sigma_a squared-hat = (50 - 10)/5 = 8 and the intraclass correlation is 8/(8 + 10) = 0.44.
The partial correlation between variables 1 and 2 given variable 3, r12.3, is the correlation between the residuals of 1 and 2 after each has been regressed on 3: it measures their linear association with the effect of 3 removed. In terms of simple correlations, r12.3 = (r12 - r13 r23) / square root of [(1 - r13 squared)(1 - r23 squared)]. Higher-order partial correlations follow recursively. A partial correlation can differ greatly from the simple correlation, and even change sign, when the third variable is related to both: yield and rainfall may correlate strongly, but the partial correlation given irrigated area may be small. Partial correlations are tested with t = r square root of (n - 2 - c) / square root of (1 - r squared), where c is the number of variables held constant, on n - 2 - c degrees of freedom.
The multiple correlation coefficient R1.23 is the correlation between variable 1 and its best linear prediction from variables 2 and 3; its square is the R squared of the regression of 1 on 2 and 3. In terms of simple correlations, R1.23 squared = (r12 squared + r13 squared - 2 r12 r13 r23) / (1 - r23 squared). The multiple correlation is never smaller than the absolute value of any simple correlation of variable 1 with the predictors, and 1 - R1.23 squared = (1 - r12 squared)(1 - r13.2 squared), which links multiple and partial correlations. In agricultural research, multiple and partial correlations are used to study yield components (how much of yield variation is explained by tillers, grains per panicle and grain weight together, and which component matters given the others), and they lead into path analysis, covered in the biometrical techniques subject.
Suppose grain yield (1), tillers per plant (2) and plant height (3) have simple correlations r12 = 0.7, r13 = 0.5 and r23 = 0.6. The partial correlation of yield with tillers given height is (0.7 - 0.5 x 0.6) / square root of (0.75 x 0.64) = 0.4 / 0.693 = 0.577, smaller than 0.7 because part of the yield-tiller association runs through height. The squared multiple correlation of yield on tillers and height is (0.49 + 0.25 - 2 x 0.7 x 0.5 x 0.6) / (1 - 0.36) = 0.32/0.64 = 0.50, so R1.23 = 0.707, only slightly higher than the simple correlation with tillers.
The raw residual e_i = y_i - y-hat_i has variance sigma squared (1 - h_ii), where h_ii is the i-th diagonal element of the hat matrix, so residuals at high-leverage points are smaller on average. The standardised residual divides e_i by s; the internally studentised residual divides by s times the square root of (1 - h_ii); the externally studentised (deleted) residual uses an estimate of sigma computed without observation i and follows a t distribution with n - p - 1 degrees of freedom under the model. Values beyond about plus or minus 2 to 3 deserve attention.
Plots of residuals are the main tool for checking a fitted model. Residuals against fitted values should show a horizontal band; a curve indicates a missing term (such as a quadratic), and a funnel indicates variance increasing with the mean, suggesting a transformation or weighted least squares. Residuals against each predictor, and against variables not in the model, reveal mis-specification. A normal quantile plot checks normality. Residuals in time or space order reveal correlation; the Durbin-Watson statistic d, approximately 2(1 - rho-hat) where rho-hat is the lag-one autocorrelation of the residuals, is near 2 for independent errors and below 2 for positive autocorrelation. When the data contain replicate observations at the same x values, the residual sum of squares can be split into pure error (variation among replicates) and lack of fit, and the lack-of-fit F test checks whether the chosen form of the model is adequate. Formal tests are aids; most problems are seen first in plots.
The leverage of observation i is h_ii, the i-th diagonal element of H. It depends only on the x values and measures how far the point lies from the centre of the predictor space. Leverages lie between 1/n and 1 in models with an intercept, and they sum to p, so their average is p/n; a common rule flags points with h_ii greater than 2p/n (or 3p/n). A high-leverage point is not necessarily bad, but it has the potential to pull the fitted surface towards itself.
Influence combines leverage with the size of the residual. Cook's distance for observation i measures how much all the fitted values change when i is deleted; it can be written D_i = (r_i squared / p) x h_ii / (1 - h_ii), where r_i is the internally studentised residual. Points with D_i near or above 1, or well above the others (some texts use 4/n), deserve scrutiny. DFFITS measures the change in the observation's own fitted value and DFBETAS the change in each coefficient, both scaled by standard errors. Influential points should be checked against the field records: a data-entry error should be corrected, a plot damaged by an identifiable cause may be excluded with a report, and a genuine extreme observation may show that the model is inadequate in that region. Deleting points simply because they are influential, without a reason, biases the analysis.
In a regression with p = 3 parameters, an observation with internally studentised residual 2.5 and leverage 0.30 has Cook's distance (2.5 squared / 3) x 0.30/0.70 = 2.083 x 0.429 = 0.89, close to 1 and worth checking. With n = 40 and p = 4, the average leverage is 0.10 and the 2p/n rule flags leverages above 0.20. An observation with raw residual 3, s = 2 and leverage 0.19 has internally studentised residual 3 / (2 x 0.9) = 1.67.
Multicollinearity is near-linear dependence among predictors, common when predictors measure related things (plant height and biomass, several weather variables, yield components). It does not bias least squares estimates, but it inflates their variances, makes individual coefficients unstable and sensitive to small data changes, can give coefficients with the wrong sign, and makes individual t tests non-significant even when the overall F is highly significant. The variance inflation factor of predictor j, VIF_j = 1 / (1 - R_j squared), where R_j squared is from regressing x_j on the other predictors, measures how much the variance of its coefficient is inflated compared with uncorrelated predictors; values above about 10 are commonly taken as a sign of serious collinearity. The eigenvalues of the correlation matrix of the predictors, and the condition number (the square root of the ratio of the largest to the smallest eigenvalue), give an overall diagnosis.
Remedies include collecting data that break the dependence, dropping or combining redundant predictors, using principal components of the predictors, and biased estimation. Ridge regression replaces (X'X)^(-1) with (X'X + kI)^(-1) on standardised predictors: beta-hat(k) = (X'X + kI)^(-1) X'y, for a small k > 0. It shrinks the coefficients towards zero, introducing bias but reducing variance, and for a suitable k its mean squared error is smaller than that of least squares. The ridge trace, a plot of coefficients against k, helps choose k at the point where the coefficients stabilise; cross-validation is the modern method. With k = 0 ridge equals least squares. The lasso, which penalises the absolute values of the coefficients, also selects variables by setting some coefficients exactly to zero; it is covered in the machine learning subject.
Orthogonal designs avoid collinearity by construction: in a balanced factorial, the contrasts for main effects and interactions are uncorrelated, so each coefficient is estimated as precisely as possible. Collinearity is mainly a problem of observational data, such as weather variables over seasons or survey data, which is one more reason to prefer designed experiments when the question allows.
When there are many candidate predictors, a subset is often chosen. Forward selection starts with no predictors and adds, one at a time, the one that most improves the fit, stopping when no addition is significant at a chosen level. Backward elimination starts with all predictors and removes the least significant one at a time. Stepwise regression combines the two, allowing a variable added earlier to be removed later. These procedures are quick but can miss the best subset, and the p-values and standard errors of the final model are too optimistic because the same data chose the model.
All-subsets regression fits every possible subset (2 to the power k models for k predictors) and compares them by a criterion. Mallows' Cp = RSS_p / s squared - n + 2p, where s squared is the residual mean square from the full model and p the number of parameters in the subset model, should be close to p for a subset with little bias. The Akaike information criterion for normal linear models can be written AIC = n ln(RSS / n) + 2p, and the Bayesian (Schwarz) criterion BIC = n ln(RSS / n) + p ln(n); smaller is better for both. BIC penalises extra parameters more heavily than AIC whenever ln(n) > 2, that is n of about 8 or more, so it tends to choose smaller models. Adjusted R squared and the prediction sum of squares (PRESS) are other criteria. No criterion replaces subject knowledge: variables central to the scientific question should usually stay in the model, and a model chosen for prediction may differ from one chosen to estimate an effect.
With n = 50, suppose model 1 has p = 3 parameters and RSS = 200, and model 2 has p = 5 and RSS = 180. AIC is 50 ln(4) + 6 = 75.3 for model 1 and 50 ln(3.6) + 10 = 74.1 for model 2, so AIC prefers the larger model. BIC is 50 ln(4) + 3 ln(50) = 81.0 for model 1 and 50 ln(3.6) + 5 ln(50) = 83.6 for model 2, so BIC prefers the smaller one. The disagreement reflects BIC's heavier penalty, 3.9 per parameter at n = 50 against AIC's 2. For a full model with residual mean square 5 and a subset model with p = 4 and RSS = 120 from n = 30, Mallows' Cp = 120/5 - 30 + 8 = 2, below p = 4, suggesting the subset has little bias.
When errors are uncorrelated but have unequal variances, V(e_i) = sigma squared / w_i, observations with smaller variance should count more. Weighted least squares minimises the sum of w_i (y_i - x_i' beta) squared, giving beta-hat = (X'WX)^(-1) X'Wy with W = diag(w_i). The weights are the reciprocals of the relative variances: if the variance is proportional to x, w = 1/x; if the standard deviation is proportional to x, w = 1/x squared; if each observation is a mean of n_i units, w = n_i. Weights usually come from the design (group sizes) or from a model of the variance estimated from residuals.
When errors are correlated or heteroscedastic with a known covariance structure, V(e) = sigma squared V, the generalised (Aitken) least squares estimator is beta-hat = (X'V^(-1) X)^(-1) X'V^(-1) y, with variance sigma squared (X'V^(-1) X)^(-1). It is the BLUE, the generalisation of the Gauss-Markov theorem; it is equivalent to transforming the data by a matrix P with P'P = V^(-1) and applying ordinary least squares. In practice V is estimated (feasible GLS), for example from an autoregressive model of the residuals in time series or from variance components in mixed models; the mixed-model estimators of the mixed-models subject are GLS with V estimated by REML. Ordinary least squares remains unbiased when V is not the identity, but its standard errors are wrong, which matters more than the loss of efficiency: inference can be badly misleading if correlation among plots or repeated measures is ignored.
In a calibration where the spread of a laboratory measurement grows in proportion to the concentration, the standard deviation is proportional to x and the weights are 1/x squared. Where each data point is a plot mean of n_i plants, weights proportional to n_i are natural. If weights are estimated from the residuals of an unweighted fit, the procedure can be iterated, but the final standard errors should be interpreted with the estimation of the weights in mind.
A model is non-linear when the response is not a linear function of the parameters, as in the exponential decay y = a exp(-bx), the Mitscherlich fertiliser response y = A[1 - exp(-c(x + b))], the logistic growth curve y = K / (1 + exp(-r(t - t0))), and the Michaelis-Menten equation. Some non-linear models are intrinsically linear: y = a x^b becomes log y = log a + b log x, and y = a exp(bx) becomes log y = log a + bx, which can be fitted by ordinary least squares on the transformed scale, although the error structure changes with the transformation.
Least squares for non-linear models minimises the residual sum of squares iteratively. The Gauss-Newton method linearises the model around current estimates with a first-order Taylor expansion and solves a linear least squares problem for the correction, repeating until convergence; the Levenberg-Marquardt method adds a damping term that makes convergence more reliable from poor starting values. Good starting values, often from a linearised fit, a graph or the biology of the parameters, are essential; the algorithm can converge to a local minimum or fail. Inference uses asymptotic results: estimated standard errors come from the linearised model, the matrix sigma squared (F'F)^(-1), where F is the matrix of derivatives of the model with respect to the parameters, and t and F tests are approximate. The quality of the approximation depends on the curvature of the model; reparameterising so that parameters have direct meaning (an asymptote, a half-life) often improves both convergence and the accuracy of intervals. Profile likelihood or bootstrap intervals are more reliable than the linearised ones when curvature is large. Growth models are developed further in the crop forecasting and growth models subject.
Fitting log y = log a + bx by ordinary least squares assumes multiplicative errors with constant variance on the log scale; fitting y = a exp(bx) directly by non-linear least squares assumes additive errors with constant variance on the original scale. The two give different estimates, and back-transforming a fitted log-scale mean gives a median rather than a mean on the original scale. The choice should follow the residual plots and the way the measurement error behaves, not convenience alone.
Students use this subject in every quantitative thesis: fitting response curves, relating yield to components or weather, analysing unbalanced experiments and checking assumptions. In examinations they must derive least squares estimates and their variances, decide estimability, apply Cochran's theorem, compute partial and multiple correlations, adjusted R squared and influence measures, and choose the right F denominator in nested and mixed models.
A five-year statistician fits and checks regression models daily: yield response to fertiliser, calibration of a quick laboratory method against a reference method, weather-yield models for forecasting, unbalanced trials with lost plots. The decisions are about specification, diagnostics, collinearity among weather variables, weighting when variances differ, and how to present uncertainty. They also review colleagues' analyses for the classic errors: wrong error term, ignored correlation, overfitted models.
A senior statistician sets analytical standards for an institution's research and statistical outputs: when generalised least squares or mixed models are required instead of ordinary least squares, how model selection must be reported, how forecasting models are validated before operational use, and which software and training the staff need.
A PhD with long experience teaches linear model theory, develops estimators and diagnostics for agricultural data (correlated field plots, heteroscedastic counts, non-linear growth processes), and judges, as reviewer and adviser, whether published conclusions rest on a model that fits.
Proofs are outlined, not given in full. REML and BLUP for mixed models, generalised linear models, robust regression and penalised methods beyond ridge are left to other subjects. Econometric topics (simultaneous equations, panel data, instrumental variables) are referred to the Agribusiness econometrics subject and this track's causal inference subject. Rules of thumb for diagnostics (VIF above 10, leverage above 2p/n, Cook's distance near 1) are conventions that differ between texts.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions. Every numerical answer was computed with a script; the values needed are stated in the question.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.