Agricultural Statistics & Data Science · Statistical theory core · also Statistical Inference (MSc core), Theory of Estimation
How to estimate well and know it: sufficiency and completeness, unbiasedness and minimum variance, the Cramer-Rao bound, the Rao-Blackwell and Lehmann-Scheffe theorems, the methods of moments, maximum likelihood and minimum chi-square, and confidence intervals.
In the exams: AICE-JRF Ag Stat Unit 1 (heavy: each theorem named); GATE ST 'Estimation' (heavy); IIT JAM MS (heavy); UPSC ISS Statistics I (heavy); ISI and CSIR-NET (present)
Estimation is the part of statistical inference that turns data into numbers for unknown quantities: a germination rate, a mean yield, a heritability, a variance component, the rate parameter of a pest-count model. A point estimator gives one value; an interval estimator gives a range with a stated confidence. The theory asks which estimators are good, in what sense, and how good the best possible estimator can be.
The subject builds on two companions. Probability Theory supplies the laws of large numbers, the central limit theorem, Slutsky's lemma and the delta method that govern large-sample behaviour. Probability Distributions and Sampling Distributions supplies the families, the exponential-family structure and the chi-square, t and F results on which exact confidence intervals rest. This manual refers to them rather than repeating them.
The chapters follow the standard master's course: properties of estimators (unbiasedness, consistency, efficiency); sufficiency and the factorisation theorem; completeness, ancillarity and Basu's theorem; minimum-variance unbiased estimation through the Rao-Blackwell and Lehmann-Scheffé theorems; the Cramér-Rao inequality and Fisher information; the methods of moments, maximum likelihood and minimum chi-square; exact and large-sample confidence intervals; and the iterative algorithms (Newton-Raphson and Fisher scoring) by which maximum-likelihood estimates are actually computed for agricultural models such as probit dose-response and genetic linkage.
Examinations test exactly these items. The ICAR AICE-JRF/SRF Agricultural Statistics paper names Cramér-Rao, Rao-Blackwell and Lehmann-Scheffé in Unit 1. GATE Statistics, IIT JAM Mathematical Statistics, the ISI tests and the Indian Statistical Service ask for UMVUEs, MLEs, information bounds and pivotal intervals as numerical and multiple-select items. In practice, every standard error in a field-trial report, every heritability with its confidence limits and every LD50 from a bioassay is an application of this subject.
Let X₁, ..., Xₙ be a random sample from a distribution with unknown parameter θ. An estimator T = T(X₁, ..., Xₙ) is any statistic used to guess θ (or a function ψ(θ)); its value for observed data is an estimate. Because T is random, its quality is judged by its sampling distribution, not by any single estimate.
T is unbiased for ψ(θ) if E_θ(T) = ψ(θ) for every θ; its bias is b(θ) = E_θ(T) − ψ(θ). The sample mean is unbiased for the population mean. The sample variance with divisor n − 1, s² = Σ(Xᵢ − X̄)²/(n − 1), is unbiased for σ², whereas the divisor-n version has expectation (n − 1)σ²/n. Unbiasedness is not preserved by non-linear functions: s is biased for σ (E(s) < σ by Jensen's inequality), even though s² is unbiased for σ². Unbiased estimators need not exist (there is none for 1/p from a single binomial observation) and can be absurd: for a Poisson count X, the only unbiased estimator of e^(−2λ) is (−1)^X, which takes negative values.
The mean squared error MSE(T) = E(T − θ)² = Var(T) + b(θ)² balances variance and bias. A biased estimator can have smaller MSE than an unbiased one: for normal samples, Σ(Xᵢ − X̄)²/(n + 1) has the smallest MSE for σ² among estimators of the form cΣ(Xᵢ − X̄)². Shrinkage estimators used in genomic prediction and small-area estimation deliberately accept bias for lower MSE.
T_n is (weakly) consistent for θ if T_n → θ in probability for every θ, and strongly consistent if the convergence is almost sure. A convenient sufficient condition: if E(T_n) → θ and Var(T_n) → 0, then T_n is consistent (mean-square consistency, via Chebyshev's inequality). By the weak law of large numbers, sample moments are consistent for population moments, and by the continuous mapping theorem continuous functions of them are consistent for the corresponding functions of population moments.
Consistency is a minimum requirement, not a mark of quality: T_n + 1000/n is consistent whenever T_n is. Unbiased estimators need not be consistent (X₁ alone is unbiased for μ but never improves), and consistent estimators need not be unbiased (the divisor-n variance). For the Cauchy location parameter the sample mean is not consistent, while the sample median is.
Consistency also depends on how the experiment grows. In a field trial, adding replicates of the same treatments makes treatment means consistent; adding new treatments with a fixed number of replicates does not make each new treatment's mean any more precise. In surveys, consistency is usually discussed as the sample size and the population size grow together, and design-consistency of an estimator is a separate requirement from model-consistency.
Among unbiased estimators, smaller variance is better. The relative efficiency of T₁ to T₂ is Var(T₂)/Var(T₁); for biased estimators the ratio of mean squared errors is used. The asymptotic relative efficiency compares limiting variances. For normal data the sample median has asymptotic variance πσ²/(2n) against σ²/n for the mean, so the efficiency of the median relative to the mean is 2/π ≈ 0.637: the median needs about 57% more observations for the same precision. For heavy-tailed data the ranking reverses.
An unbiased estimator whose variance attains the Cramér-Rao lower bound (Chapter 5) is called efficient, and its efficiency is the ratio of the bound to its variance. A uniformly minimum-variance unbiased estimator (UMVUE) has the smallest variance among all unbiased estimators at every θ; it may or may not attain the Cramér-Rao bound. Chapter 4 shows how to find it.
A worked example: for X₁, ..., Xₙ uniform on (0, θ), the moment estimator 2X̄ is unbiased with variance θ²/(3n), and (n + 1)X₍ₙ₎/n is unbiased with variance θ²/[n(n + 2)]. For n = 10 the ratio is 12/3 = 4: the estimator based on the maximum is four times as efficient, and the advantage grows with n.
A statistic T is sufficient for θ if the conditional distribution of the sample given T does not depend on θ. Intuitively T captures all the information about θ in the data; once T is known, the rest of the sample is noise that could be regenerated by a random device without knowing θ. For n Bernoulli trials, given the number of successes T = ΣXᵢ = t, every arrangement of the t successes is equally likely, with probability 1/C(n, t), whatever p is: T is sufficient.
Checking the definition directly is tedious. The Neyman-Fisher factorisation theorem gives a practical test: T is sufficient for θ if and only if the joint density or mass function factorises as f(x₁, ..., xₙ; θ) = g(T(x); θ) h(x), where h does not involve θ. Examples: Poisson, L = e^(−nλ)λ^(Σxᵢ)/Πxᵢ!, so ΣXᵢ is sufficient; normal with both parameters unknown, (ΣXᵢ, ΣXᵢ²), equivalently (X̄, s²); exponential with rate λ, ΣXᵢ; gamma with both parameters, (ΣXᵢ, ΣlogXᵢ); uniform on (0, θ), L = θ⁻ⁿ 1{X₍ₙ₎ ≤ θ}, so the maximum X₍ₙ₎ is sufficient; uniform on (θ₁, θ₂), the pair (X₍₁₎, X₍ₙ₎).
Note the indicator function in the uniform case: when the support depends on θ, it must be kept inside g, which is why order statistics become sufficient. Any one-to-one function of a sufficient statistic is sufficient, and the whole sample (or the set of order statistics) is always sufficient, trivially. For the exponential family f = h(x)exp[η(θ)T(x) − A(θ)], ΣT(Xᵢ) is sufficient (see the distributions manual).
Because the full sample is always sufficient, we want the greatest possible reduction. A sufficient statistic T is minimal sufficient if it is a function of every other sufficient statistic. The Lehmann-Scheffé criterion identifies it: if, for every pair of sample points x and y, the ratio f(x; θ)/f(y; θ) is free of θ exactly when T(x) = T(y), then T is minimal sufficient.
Applying the criterion: for N(μ, σ²) the ratio is free of (μ, σ²) iff Σxᵢ = Σyᵢ and Σxᵢ² = Σyᵢ², so (ΣX, ΣX²) is minimal. For uniform (θ, θ + 1) the minimal sufficient statistic is (X₍₁₎, X₍ₙ₎), two-dimensional for a one-dimensional parameter. For the Cauchy location family, and for the Laplace (double exponential) location family, no reduction below the full set of order statistics is possible. In a k-parameter exponential family of full rank, the natural statistic (ΣT₁(Xᵢ), ..., ΣT_k(Xᵢ)) is minimal sufficient.
Sufficiency has a practical meaning for an agricultural statistician: an analysis that depends on the data only through a minimal sufficient statistic loses nothing. Germination trials can be summarised by the count germinated, Poisson pest counts by their total, normal yields by their mean and sum of squares; storing those summaries alone is enough for any analysis under the model. If the model is wrong, though, the discarded detail is exactly what is needed to detect it, which is why residuals are kept.
A statistic T, or its family of distributions, is complete if E_θ[g(T)] = 0 for all θ implies g(T) = 0 with probability one for all θ. It is boundedly complete if this holds for bounded g. Completeness says that T has no non-trivial function with mean zero under every θ: it cannot be used to build a 'zero estimator', so there is at most one unbiased estimator of any ψ(θ) that is a function of T.
Examples. For n Bernoulli trials, T = ΣXᵢ is binomial(n, p): E g(T) = Σ g(t)C(n, t)pᵗ(1 − p)ⁿ⁻ᵗ is a polynomial in p/(1 − p) whose vanishing on an interval forces every g(t) = 0, so T is complete. For uniform (0, θ), X₍ₙ₎ is complete: E g(X₍ₙ₎) = ∫₀^θ g(t) n tⁿ⁻¹/θⁿ dt = 0 for all θ forces g = 0 almost everywhere by differentiating in θ. For a full-rank exponential family whose natural parameter space contains an open set, the natural sufficient statistic is complete.
Counterexamples. For uniform (θ, θ + 1), (X₍₁₎, X₍ₙ₎) is minimal sufficient but not complete, because the range X₍ₙ₎ − X₍₁₎ has a distribution free of θ and so its centred version has zero mean for all θ. For N(θ, θ²) (a curved exponential family) the minimal sufficient statistic (ΣX, ΣX²) is not complete. Minimal sufficiency does not imply completeness; complete sufficiency does imply minimal sufficiency (when a minimal sufficient statistic exists).
A statistic V is ancillary if its distribution does not depend on θ. Alone it says nothing about θ, but it can describe the precision of an experiment. Location families make ancillaries easy: for X₁, ..., Xₙ from f(x − θ), the range and the differences Xᵢ − X̄ are ancillary. For N(μ, σ²) with σ² known, s² is ancillary for μ. For scale families, ratios such as X₁/Xₙ are ancillary.
Basu's theorem: if T is boundedly complete and sufficient for θ, and V is ancillary, then T and V are independent for every θ. Its power is that it proves independence without computing any joint distribution. Classic applications: for N(μ, σ²) with σ² fixed, X̄ is complete sufficient for μ and s² is ancillary, so X̄ and s² are independent, the result underlying Student's t; since this holds for every σ², it holds unconditionally. For uniform (0, θ), X₍ₙ₎ is complete sufficient and X₍₁₎/X₍ₙ₎ is ancillary (scale invariant), so they are independent, which gives E(X₍₁₎) = E(X₍₁₎/X₍ₙ₎)E(X₍ₙ₎). For exponential samples, ΣXᵢ is independent of X₁/ΣXᵢ.
Without completeness the conclusion can fail: in the uniform (θ, θ + 1) case the minimal sufficient statistic contains the ancillary range itself. Fisher argued that inference should be conditional on ancillary statistics, a principle that remains debated, and that underlies conditional inference for 2 × 2 tables and regression on fixed covariates.
Rao-Blackwell theorem: let T be sufficient for θ and U any unbiased estimator of ψ(θ). Then φ(T) = E(U | T) is a statistic (free of θ, because T is sufficient), it is unbiased for ψ(θ), and Var(φ(T)) ≤ Var(U) for every θ, with equality only if U is already a function of T. The proof is the law of total variance: Var(U) = Var[E(U | T)] + E[Var(U | T)].
The theorem tells us never to use an estimator that is not a function of a sufficient statistic: conditioning can only help. The same holds for any convex loss, not just squared error. A standard example: for Poisson counts, U = 1{X₁ = 0} is unbiased for e^(−λ) = P(X = 0). Given T = ΣXᵢ = t, X₁ is binomial(t, 1/n), so E(U | T = t) = (1 − 1/n)ᵗ. The Rao-Blackwellised estimator ((n − 1)/n)^T is unbiased for e^(−λ) and far less variable than the indicator of a single plot.
Lehmann-Scheffé theorem: if T is complete and sufficient, and φ(T) is unbiased for ψ(θ), then φ(T) is the unique UMVUE of ψ(θ). Combined with Rao-Blackwell, this gives two routes. Route 1: guess a function of the complete sufficient statistic and adjust it to be unbiased. Route 2: start from any simple unbiased estimator and condition on the complete sufficient statistic.
Standard results. Bernoulli: X̄ is the UMVUE of p; the UMVUE of p(1 − p) is nX̄(1 − X̄)/(n − 1); the UMVUE of p² is T(T − 1)/[n(n − 1)] with T = ΣXᵢ. Poisson: X̄ for λ, T(T − 1)/n² for λ², ((n − 1)/n)^T for e^(−λ), and (T/n)(1 − 1/n)^(T−1) for λe^(−λ) = P(X = 1). Normal with both parameters unknown: X̄ for μ, s² for σ², and c_n s for σ with the constant c_n = √((n − 1)/2) Γ((n − 1)/2)/Γ(n/2) that removes the bias of s. Uniform (0, θ): (n + 1)X₍ₙ₎/n. Exponential with mean θ: X̄; with rate λ, (n − 1)/ΣXᵢ (not 1/X̄, which is biased upward by the factor n/(n − 1)).
Two cautions. A UMVUE exists only if some unbiased estimator exists. And the UMVUE is optimal only within unbiased estimators: for e^(−2λ) in the single-observation Poisson case, the UMVUE (−1)^X is useless, and a biased estimator such as e^(−2X̄) is far better in MSE. In applied work UMVUEs matter most for survey estimators of totals and proportions and for small-sample estimation of variances, where unbiasedness is a formal requirement.
For a density f(x; θ) the score is S(θ) = ∂ log f(X; θ)/∂θ. Under regularity conditions (the support does not depend on θ and differentiation under the integral sign is allowed), E_θ[S(θ)] = 0 and the Fisher information in one observation is I(θ) = E[S(θ)²] = −E[∂² log f/∂θ²]. For a random sample the information adds: I_n(θ) = nI(θ).
Standard values per observation: Bernoulli(p), 1/[p(1 − p)]; Poisson(λ), 1/λ; N(μ, σ²) for μ, 1/σ²; N(μ, σ²) for σ² with μ known, 1/(2σ⁴); exponential with rate λ, 1/λ²; exponential with mean θ, 1/θ². Information depends on the parameterisation: if η = g(θ), I(η) = I(θ)/[g′(θ)]². For several parameters the information matrix has entries −E[∂² log f/∂θᵢ∂θⱼ]; its inverse gives the lower bounds and the asymptotic covariance of maximum-likelihood estimates.
Why does information measure precision? The score is the slope of the log-likelihood. If the log-likelihood is sharply curved near the true value, small changes in θ produce large changes in its slope, data pin θ down tightly, and the information is large. A flat log-likelihood means that a range of θ values explains the data almost equally well. The second-derivative form −E(∂²log f/∂θ²) expresses exactly this curvature. For a binomial proportion the information 1/[p(1 − p)] is smallest at p = 1/2 and grows without bound near 0 and 1, which is why a rare germination failure rate is estimated with small absolute error but large relative error.
Cramér-Rao inequality: under the regularity conditions, any unbiased estimator T of ψ(θ) from a sample of n satisfies Var(T) ≥ [ψ′(θ)]²/[nI(θ)]. For ψ(θ) = θ the bound is 1/[nI(θ)]. For a biased estimator with bias b(θ), the bound becomes [ψ′(θ) + b′(θ)]²/[nI(θ)]. The proof is the Cauchy-Schwarz inequality applied to Cov(T, S) = ψ′(θ).
Examples: for Poisson counts the bound for λ is λ/n, attained by X̄; for Bernoulli trials the bound for p is p(1 − p)/n, attained by X̄; for a normal mean, σ²/n, attained by X̄; for σ² of a normal with μ known, 2σ⁴/n, attained by Σ(Xᵢ − μ)²/n. With μ unknown, the UMVUE s² has variance 2σ⁴/(n − 1), slightly above the bound 2σ⁴/n: a UMVUE need not attain the bound.
Equality holds if and only if the score is linear in T: ∂ log L/∂θ = k(θ)[T − ψ(θ)]. This happens only in one-parameter exponential families and only for ψ(θ) = E(T), the mean of the natural sufficient statistic. So, for the exponential distribution with mean θ, X̄ attains the bound θ²/n for θ, but no unbiased estimator of the rate λ = 1/θ attains its bound λ²/n; the UMVUE (n − 1)/ΣX has variance λ²/(n − 2).
When the support depends on θ the theorem does not apply. For uniform (0, θ), (n + 1)X₍ₙ₎/n has variance θ²/[n(n + 2)], of order 1/n², smaller than any 'bound' of order 1/n that a naive calculation suggests. Sharper bounds for non-attaining cases exist (Bhattacharyya bounds using higher derivatives; the Chapman-Robbins bound, which needs no differentiability).
Fisher information is a design tool. In a dose-response bioassay the information about the LD50 is greatest when doses are placed near the expected LD50, which is why pilot trials are used to locate it before the main experiment. In genetic linkage studies the information per individual about a recombination fraction depends on the cross: a backcross gives more per individual than an F₂ with dominant markers. In sample surveys and field trials, comparing information per unit cost compares designs. And observed information, the negative second derivative of the log-likelihood at the estimate, gives the standard errors printed by every maximum-likelihood program.
Information also explains the cost of grouping and coarsening data. When yields are recorded only as 'above or below a threshold', or pest counts only as 'infested or not', the information per observation falls, sometimes sharply; the ratio of grouped to ungrouped information is the efficiency lost by the cruder recording. Before deciding to score disease on a 0-5 scale rather than measure lesion area, a statistician can compare the information per plant under each recording method against the cost of recording.
The method of moments, introduced by Karl Pearson, equates the first k sample moments mᵣ′ = (1/n)ΣXᵢʳ to the corresponding population moments, expressed in terms of the k unknown parameters, and solves. Central moments can be used instead of raw moments.
Examples. Poisson: λ̂ = X̄. Uniform (0, θ): E(X) = θ/2, so θ̂ = 2X̄. Normal: μ̂ = X̄, σ̂² = (1/n)Σ(Xᵢ − X̄)². Gamma with shape α and rate β: from mean α/β and variance α/β², α̂ = X̄²/S² and β̂ = X̄/S², where S² is the sample variance. Beta(a, b): solving the mean and variance equations gives â = X̄[X̄(1 − X̄)/S² − 1] and b̂ = (1 − X̄)[X̄(1 − X̄)/S² − 1]. Negative binomial for clumped pest counts: k̂ = X̄²/(S² − X̄), valid only when S² > X̄. Binomial with both n and p unknown, as when the number of insects exposed is unknown: n̂ = X̄²/(X̄ − S²) and p̂ = X̄/n̂, which behaves badly when S² is close to X̄.
Moment estimators are easy to compute, need no distributional form beyond the moments, and are consistent whenever the moments exist and the parameters are continuous functions of them (law of large numbers plus continuous mapping). Their asymptotic normality follows from the multivariate central limit theorem and the delta method.
They are often inefficient. For the uniform (0, θ), 2X̄ has variance θ²/(3n) against θ²/[n(n + 2)] for the estimator based on the maximum. For the gamma shape and for the negative binomial k, maximum likelihood is considerably more efficient when the shape or k is small. Moment estimates can fall outside the parameter space: 2X̄ can be smaller than the largest observation, an impossible value for θ; k̂ is negative when S² < X̄; and n̂ for the binomial can be negative or absurdly large. Moment estimates are also not functions of the minimal sufficient statistic in general.
Their best use today is as starting values for iterative maximum likelihood (Chapter 11), as quick estimates in field reports, and in models where the likelihood is intractable but moments are easy, the idea behind the generalised method of moments of econometrics.
The likelihood L(θ) = Πf(xᵢ; θ) is the joint density viewed as a function of θ for fixed data. The maximum-likelihood estimate (MLE) θ̂ is the value that maximises it, usually found by solving the likelihood equation ∂ log L/∂θ = 0 and checking that it is a maximum, but sometimes on the boundary.
Examples. Bernoulli: p̂ = X̄. Poisson: λ̂ = X̄. Normal: μ̂ = X̄, σ̂² = (1/n)Σ(Xᵢ − X̄)² (biased; the factor (n − 1)/n). Exponential with rate λ: λ̂ = 1/X̄. Uniform (0, θ): L = θ⁻ⁿ for θ ≥ X₍ₙ₎ and 0 otherwise, decreasing in θ, so θ̂ = X₍ₙ₎, found without calculus; it is biased, with mean nθ/(n + 1). Laplace (double exponential) location: the MLE is the sample median. Uniform (θ − 1/2, θ + 1/2): any value between X₍ₙ₎ − 1/2 and X₍₁₎ + 1/2 maximises L, so the MLE is not unique. For the Cauchy location model the likelihood equation can have several roots, and the global maximum must be found numerically.
Invariance: if θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ) for any function g (for a non-one-to-one g, using the induced likelihood). So the MLE of σ is the square root of the MLE of σ², the MLE of P(X = 0) = e^(−λ) for Poisson counts is e^(−X̄), and the MLE of a heritability ratio is the ratio of the MLEs of the variance components. Unbiasedness has no such property.
The MLE is always a function of the minimal sufficient statistic (when it is unique), but it need not be unbiased, and in small samples it need not be best in mean squared error. In variance-component estimation the ML estimates are biased downward because they ignore the degrees of freedom used by fixed effects; restricted maximum likelihood (REML), standard in plant and animal breeding, corrects this, just as the divisor n − 1 corrects the normal variance.
Under regularity conditions (identifiability, a fixed support, smoothness of the log-likelihood, a true value in the interior of the parameter space), the MLE is consistent, and √n(θ̂ − θ) → N(0, 1/I(θ)) in distribution: it is asymptotically unbiased, asymptotically normal and asymptotically efficient, attaining the Cramér-Rao bound in the limit. Estimators with this property are called best asymptotically normal (BAN). For several parameters the asymptotic covariance is the inverse information matrix I(θ)⁻¹/n. Combined with the delta method, the asymptotic variance of g(θ̂) is [g′(θ)]²/[nI(θ)].
In practice I(θ) is replaced by I(θ̂) (expected information) or by the observed information −∂² log L/∂θ² at θ̂; both give consistent standard errors, and Efron and Hinkley argued that observed information is often preferable. The regularity conditions matter: for the uniform (0, θ), n(θ − X₍ₙ₎)/θ tends to a standard exponential distribution, not a normal, and the convergence rate is 1/n rather than 1/√n. When the true value lies on the boundary of the parameter space, as when a variance component is zero, the limiting distribution is a mixture and standard errors mislead. When the number of parameters grows with n, as in the Neyman-Scott problem of many pairs each with its own mean, the MLE of the common variance is inconsistent, converging to σ²/2.
For grouped or categorical data with observed frequencies Oⱼ and expected frequencies Eⱼ(θ) = Npⱼ(θ), the minimum chi-square estimator minimises Pearson's χ² = Σ[Oⱼ − Eⱼ(θ)]²/Eⱼ(θ). The modified minimum chi-square estimator (Neyman) uses the observed frequency in the denominator, Σ[Oⱼ − Eⱼ(θ)]²/Oⱼ, which often makes the estimating equations linear and easy to solve; it fails when any Oⱼ is zero. Minimising the likelihood-ratio statistic G² = 2ΣOⱼ log(Oⱼ/Eⱼ) is the same as maximum likelihood for multinomial data.
Under regularity all three are BAN estimators: asymptotically equivalent to the MLE from the grouped data, and when θ is estimated by any of them the goodness-of-fit statistic is asymptotically χ² with (number of classes − 1 − number of parameters) degrees of freedom. In small samples they differ. Minimum chi-square methods were long used in genetics (estimating linkage from F₂ classes) and in bioassay, where Berkson's minimum logit chi-square gave quick, nearly efficient estimates of logistic dose-response parameters before computers made maximum likelihood routine.
Least squares minimises Σ[Xᵢ − E(Xᵢ; θ)]²; in the linear model it gives the best linear unbiased estimators by the Gauss-Markov theorem (treated in Linear Models and Regression), and under normal errors it coincides with maximum likelihood. Weighted least squares handles unequal variances.
A Bayes estimator minimises expected posterior loss: under squared-error loss it is the posterior mean, under absolute-error loss the posterior median, and under 0-1 loss the posterior mode. With a beta(a, b) prior and t successes in n Bernoulli trials, the posterior mean is (t + a)/(n + a + b), a compromise between the sample proportion and the prior mean that shrinks small-sample estimates towards the prior. Bayes estimators are usually biased, are generally admissible, and are the basis of empirical-Bayes and BLUP methods (Bayesian Inference covers them in depth).
Other methods include the method of percentiles (equating sample and population quantiles, useful for Weibull fits to failure data), M-estimators that minimise a robust loss such as Huber's, L-estimators such as trimmed means that are linear combinations of order statistics, and minimum-distance estimators. Estimators may also be judged by equivariance: the Pitman estimator is the best location-equivariant estimator under squared error.
A confidence interval [L(X), U(X)] for θ has coverage probability P_θ(L ≤ θ ≤ U) for each θ; its confidence coefficient is the infimum of the coverage over θ. A 95% confidence interval is a procedure that captures the true θ in 95% of repeated samples. Once data are observed, a particular interval either contains θ or does not; the 95% describes the method, not the posterior probability for that interval (a Bayesian credible interval has that meaning instead). Among intervals with the same confidence, shorter ones (in expected length) are preferred; for discrete data exact coverage at every θ is impossible without randomisation, and 'exact' intervals such as Clopper-Pearson guarantee at least the nominal coverage.
Confidence intervals can be one-sided. A seed company may need only a lower confidence limit for germination ('at least 85% with 95% confidence'), and a residue regulator only an upper limit. One-sided limits use the full α in one tail: z₀.₀₅ = 1.645 instead of z₀.₀₂₅ = 1.96 for a normal pivot. Simultaneous intervals for several parameters (Bonferroni, Scheffé, Tukey) keep the joint coverage at the stated level and are discussed with multiple comparisons in the design subjects.
A pivot is a function Q(X, θ) whose distribution does not depend on θ. If P(a ≤ Q ≤ b) = 1 − α and Q is monotone in θ, inverting the inequality gives a confidence interval.
Standard pivots for normal samples (their distributions are derived in the distributions manual): (X̄ − μ)/(σ/√n) ~ N(0, 1) with σ known, giving X̄ ± z_(α/2)σ/√n; (X̄ − μ)/(s/√n) ~ t_(n−1), giving X̄ ± t_(α/2, n−1)s/√n; (n − 1)s²/σ² ~ χ²_(n−1), giving the interval [(n − 1)s²/χ²_(α/2), (n − 1)s²/χ²_(1−α/2)] for σ², where χ²_(α/2) is the upper α/2 point; (s₁²/σ₁²)/(s₂²/σ₂²) ~ F(n₁ − 1, n₂ − 1) for a variance ratio; and the two-sample t pivot for μ₁ − μ₂ with equal variances. For exponential lifetimes with rate λ, 2λΣXᵢ ~ χ²_(2n), giving an interval for λ or the mean life. For uniform (0, θ), X₍ₙ₎/θ has CDF tⁿ on (0, 1), giving the interval [X₍ₙ₎, X₍ₙ₎/α^(1/n)] of confidence 1 − α.
Choosing a and b: for a symmetric unimodal pivot (normal, t), equal tails give the shortest interval. For skewed pivots (chi-square, F) equal tails are conventional but not shortest; the shortest interval for σ² requires unequal tails satisfying a density condition, and the gain is small for moderate n. For the uniform case, the one-sided choice above is the shortest, because the pivot's density increases towards 1.
When no exact pivot exists, intervals can be built by inverting a family of tests (Chapter 8 of the hypothesis-testing manual), from the likelihood, by large-sample approximation (Chapter 10) or by the bootstrap.
Ten plots of a new wheat variety give a mean yield of 4.2 t/ha with s = 0.5 t/ha. With t_(0.025, 9) = 2.262, the 95% confidence interval for the mean is 4.2 ± 2.262 × 0.5/√10 = 4.2 ± 0.358, that is 3.84 to 4.56 t/ha. The interval for σ² uses χ²_9 points 19.023 and 2.700: [9 × 0.25/19.023, 9 × 0.25/2.700] = [0.118, 0.833], so σ lies between 0.34 and 0.91 t/ha. The width of the variance interval, almost three-fold in σ, is a reminder that ten plots estimate a mean far better than they estimate variability.
The same data give a prediction interval for the yield of one further plot: 4.2 ± 2.262 × 0.5 × √(1 + 1/10) = 4.2 ± 1.19, that is 3.0 to 5.4 t/ha. It is much wider than the confidence interval for the mean, because it must cover the plot-to-plot variability as well as the uncertainty in the mean. Confusing the two is a common error when trial results are passed to extension staff: a farmer's single field behaves like one new plot, not like the trial mean.
When θ̂ is the MLE, its asymptotic normality gives the Wald interval θ̂ ± z_(α/2)/√(nI(θ̂)), or θ̂ ± z_(α/2) × SE using observed information. For a proportion, p̂ ± z√(p̂(1 − p̂)/n); for a Poisson rate, X̄ ± z√(X̄/n); for any smooth g, the delta method gives g(θ̂) ± z|g′(θ̂)|SE(θ̂).
The Wald interval for a proportion is known to have poor coverage when n is small or p is near 0 or 1: coverage oscillates and can fall well below 95%, and when p̂ = 0 the interval collapses to a point. Agresti and Coull's comparison made this widely known.
Wald intervals for other parameters inherit similar problems whenever the log-likelihood is far from quadratic: for a Poisson rate with small counts, a ratio of means, a variance component, or a heritability near 0 or 1, the symmetric interval can extend into impossible values (negative rates or variances, heritabilities above 1). Truncating such an interval at the boundary hides the problem rather than solving it. The remedies are the transformed, score and likelihood-ratio intervals below, or exact methods based on the binomial, Poisson or chi-square distributions.
The Wilson (score) interval inverts the score test: it solves |p̂ − p| ≤ z√(p(1 − p)/n) for p, giving [p̂ + z²/(2n) ± z√(p̂(1 − p̂)/n + z²/(4n²))]/(1 + z²/n). Its coverage is close to nominal even for small n, and it never leaves [0, 1]. The Agresti-Coull interval adds two successes and two failures (for 95%) and then uses the Wald form. Likelihood-ratio intervals {θ : 2[log L(θ̂) − log L(θ)] ≤ χ²_(1, α)} respect the shape of the likelihood and are invariant to reparameterisation.
Transformation helps too. Compute the interval on a scale where the estimator is closer to normal with stable variance and back-transform: the log scale for rates and ratios, the logit or arcsine scale for proportions, and Fisher's z = (1/2)log[(1 + r)/(1 − r)] for a correlation coefficient, which is approximately normal with mean (1/2)log[(1 + ρ)/(1 − ρ)] and variance 1/(n − 3). For r = 0.6 from n = 28 plots, z = 0.693, SE = 0.2, the 95% interval for z is 0.301 to 1.085, and back-transforming (r = tanh z) gives 0.29 to 0.80 for ρ: asymmetric about 0.6, as it should be.
Large-sample formulas give sample sizes. To estimate a mean within ±d with 95% confidence, n ≈ (1.96σ/d)². To estimate a proportion within ±d, n ≈ 1.96²p(1 − p)/d², at most 1.96²/(4d²) ≈ 385 for d = 0.05. For relative precision, n ≈ (1.96 CV/r)², with r the allowable relative error: estimating a mean yield with CV 20% to within ±5% needs about 62 plots. These formulas assume simple random sampling; survey designs adjust them by the design effect.
Three practical refinements are common. When σ is unknown and estimated from the same small sample, the t multiplier replaces 1.96 and the calculation is iterated (start with z, compute n, replace z by t with n − 1 df, recompute). When the population is finite and the sample a large fraction of it, the finite population correction reduces n: n' = n/(1 + n/N). And when the target is a difference between two means rather than one mean, the variance doubles, so each group needs about twice the single-sample size; power considerations, treated in the hypothesis-testing manual, raise it further.
Most agricultural likelihoods have no closed-form maximum: probit and logit dose-response models, gamma and Weibull shapes, negative binomial k, variance components, linkage with incomplete classification. The MLE is then found by iteration.
Newton-Raphson expands the score about the current value: θ₍ₖ₊₁₎ = θ₍ₖ₎ − ℓ′(θ₍ₖ₎)/ℓ″(θ₍ₖ₎), where ℓ = log L; in several dimensions θ₍ₖ₊₁₎ = θ₍ₖ₎ + [−ℓ″(θ₍ₖ₎)]⁻¹ℓ′(θ₍ₖ₎), with ℓ″ the Hessian matrix. Near the maximum convergence is quadratic: the number of correct digits roughly doubles each step. Far from it, Newton-Raphson can diverge or converge to a minimum or a saddle, especially when the log-likelihood is not concave (as for Cauchy location). Good starting values, usually moment estimates, and step-halving when the likelihood fails to increase make it reliable.
A worked one-step example: for the Cauchy location model with scale 1, the score is Σ 2(xᵢ − θ)/[1 + (xᵢ − θ)²]. Starting from the sample median, one or two Newton steps typically settle on the MLE, but starting from a distant value can lead to a different root.
Fisher's method of scoring replaces the observed information −ℓ″(θ) by the expected information nI(θ): θ₍ₖ₊₁₎ = θ₍ₖ₎ + ℓ′(θ₍ₖ₎)/[nI(θ₍ₖ₎)]. Expected information is always positive (definite), so each step moves uphill for small enough step size, and it is often simpler to compute; convergence is linear rather than quadratic. For exponential families in their canonical parameterisation, observed and expected information coincide and the two methods are identical: for a logistic regression, scoring and Newton-Raphson give the same iterations, which take the form of iteratively reweighted least squares (IRLS), the algorithm used to fit all generalised linear models. Fisher developed scoring for exactly the problems of agricultural genetics and bioassay, and Finney's probit analysis computed it by hand with working probits and weights.
A useful theoretical result: starting from any √n-consistent estimator, a single Newton-Raphson or scoring step gives an estimator that is asymptotically efficient (as good as the MLE to first order). This justifies one-step estimators.
The EM algorithm (Dempster, Laird and Rubin) handles missing or latent data by alternating an expectation step, filling in the expected complete-data log-likelihood, and a maximisation step; it increases the likelihood at every iteration but converges slowly. Classic agricultural examples are the estimation of gene frequencies in the ABO-type blood-group problem, missing plots, mixtures and variance components. Quasi-Newton methods approximate the Hessian from successive gradients.
Whatever the algorithm, the practitioner should check convergence with more than one criterion (change in parameters, change in log-likelihood and size of the score), try several starting values when multiple maxima are possible, confirm that the Hessian at the solution is negative definite, and watch for estimates drifting to the boundary, such as a variance component going to zero or a probability to 0 or 1, where standard errors from the information matrix are invalid.
A short worked case: fitting a probit dose-response line to mortality of an insect pest at five insecticide doses. Starting values come from an unweighted straight line through the empirical probits; each scoring iteration computes working probits and weights from the current line, fits a weighted regression and updates the intercept and slope; three or four iterations usually suffice. The LD50 is then −intercept/slope on the log-dose scale, with a standard error from the delta method or, better, Fieller's theorem, which gives asymmetric limits when the slope is poorly determined.
A student meets estimation as the theoretical core of the first-year master's course and as one of the highest-yield topics in AICE-JRF, GATE ST and IIT JAM. The concrete skills are: writing a likelihood and finding the MLE, including boundary and non-regular cases; finding a sufficient statistic by factorisation and deciding whether it is complete; producing a UMVUE by Lehmann-Scheffé or by conditioning; computing Fisher information and the Cramér-Rao bound; and building an exact interval from a pivot. A thesis needs the same skills to justify standard errors and confidence limits for every estimate it reports.
A statistician five years into work at a research institute estimates parameters daily: LD50s and their fiducial or confidence limits from bioassays by iterative probit fitting, heritabilities and variance components by REML, negative binomial k for pest sampling plans, proportions and rates with Wilson or exact intervals rather than Wald intervals when counts are small, and the sample size needed for a stated precision. Recognising when software output is unreliable (non-convergence, estimates at the boundary, standard errors from a flat likelihood) is part of the job.
A senior statistician with twenty-five years of experience sets estimation standards for an institute or a national programme: which estimators of crop yield, area and production are official, how their standard errors are computed and published, when model-based or shrinkage estimators may replace design-unbiased ones, and how bias-variance trade-offs are explained to policy users. Such a statistician reviews methods sections, insists that interval estimates accompany point estimates and judges whether a new estimator's claimed efficiency survives realistic departures from its model.
A PhD with long experience develops new estimators and proves their properties: sufficiency reductions and UMVUEs for new designs, efficiency bounds for semi-parametric and mixed models, asymptotic normality of estimators for dependent spatial field data, and algorithms (EM, scoring, Monte Carlo) for genetic and ecological likelihoods. Such a researcher also teaches the subject to doctoral students and judges, in refereeing, whether asymptotic claims hold at the sample sizes agricultural experiments can afford.
Proofs are sketched rather than given in full. Not covered: decision-theoretic admissibility and minimaxity beyond brief mention, equivariant estimation in depth, the James-Stein phenomenon, semi-parametric efficiency, robust estimation theory, the bootstrap and jackknife in detail, fiducial inference, and Bayesian estimation beyond the posterior-mean idea (Bayesian Inference covers it). The linear-model theory of best linear unbiased estimation is in Linear Models and Regression. Tables of t and chi-square percentage points are not reproduced; the values used are stated where needed.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions. Every numerical answer was computed with a script; the values needed are stated in the question.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.