Agricultural Statistics & Data Science · Sample surveys and official agricultural statistics · also Sampling Techniques, Sample Surveys I, Basic Sampling Techniques (minor course)
Probability sampling from first principles: sampling versus complete enumeration, simple random sampling with and without replacement, stratification and allocation, systematic sampling, and the ratio and regression estimators that use auxiliary information such as last year's area.
In the exams: AICE-JRF Ag Stat Unit 5 (heavy); UPSC ISS Statistics III (heavy); SSC CGL JSO (heavy); ASRB NET Ag Stat (heavy); state statistical-officer exams (heavy)
Sampling theory is the mathematics of learning about a whole population from a part of it, and agriculture is where much of it was built: crop-cutting surveys in India under Mahalanobis and Sukhatme, the area-frame surveys of the United States Department of Agriculture, and the agricultural censuses and sample surveys promoted by FAO worldwide. This first sampling subject covers the foundations that every survey statistician uses: why and when to sample rather than enumerate everything, probability sampling and frames, simple random sampling with and without replacement, estimation of proportions and sample size, stratified sampling with its allocation methods and the construction and collapsing of strata, systematic sampling, and the ratio, regression, difference and product estimators that use auxiliary information.
The emphasis is on what each design and estimator does, the formulas for estimates and their variances, and the conditions under which one method beats another. Those conditions are what examinations ask about and what practitioners need when they choose a design for a crop survey or a livestock census follow-up.
The subject is examined heavily. It is the first half of Unit 5 Sample Surveys in the ICAR AICE-JRF/SRF Agricultural Statistics syllabus; sampling techniques are a staple of the Indian Statistical Service papers (as reported), the statistics paper for SSC Junior Statistical Officer posts, state statistical-officer examinations and ASRB NET in agricultural statistics; and the AIEEA-PG Group 07 statistics unit introduces random sampling and standard errors. Numerical questions usually ask for a variance, a standard error, a sample size, an allocation, or the efficiency of one method over another, and this manual's question bank works those numbers.
Unequal-probability sampling, cluster and multistage designs, double sampling, sampling on successive occasions, non-response and imputation are in Sampling Theory II; small-area estimation, crop-cutting surveys and survey operations have their own subjects.
A population, in sampling theory, is the finite collection of units about which information is wanted: all farm holdings in a district, all fields sown to wheat this season, all villages in a state, all dairy cooperatives. Each unit carries values of the characters of interest, such as area under a crop, yield, number of cattle or household income. The population quantities to be estimated, called parameters, are usually totals, means, proportions and ratios: the total wheat area, the mean yield per hectare, the proportion of holdings using certified seed, or the ratio of irrigated to cultivated area. A complete enumeration, or census, observes every unit. A sample survey observes a part of the population, chosen by a stated procedure, and uses it to estimate the parameters. Censuses remain essential where a value is needed for every unit or every small area (land records, the population census, the agricultural census of holdings in many countries) and they supply the frames from which later samples are drawn. For most continuing statistics, however, countries use samples: crop yields are estimated from crop-cutting experiments on sampled fields, livestock products from sample surveys, and farm-household conditions from national sample surveys.
Sampling has four classical advantages. It costs less, because fewer units are enumerated. It is quicker, so estimates arrive while they can still be used, for example before procurement or trade decisions. It can be more accurate, because a smaller field staff can be better trained and supervised and the data better checked, so non-sampling errors are smaller; a census, with its huge staff, often has larger measurement and processing errors. And it is the only option when measurement destroys the unit, as in harvesting a plot for yield or testing seed for germination. Its limitation is that results are subject to sampling error, which is large when the population is broken down into small domains. That is why national surveys give reliable state or zone estimates but often not reliable block or village estimates, and why methods such as small-area estimation were developed.
The total error of a survey estimate has two parts. Sampling error arises because only a part of the population is observed; it would vanish in a complete enumeration done without other mistakes. For probability samples it can be estimated from the sample itself, and it falls as sample size grows, roughly in proportion to one over the square root of n. Non-sampling errors arise from everything else: defects in the frame (coverage errors such as missing or duplicated units), failure to obtain data from selected units (non-response), wrong answers or wrong measurements (response or measurement error, including recall bias and interviewer effects), and errors in coding, data entry and tabulation (processing errors). They occur in censuses and samples alike, they do not necessarily fall as n increases and they often grow with the scale of the operation, and they usually cause bias rather than random scatter. The mean squared error, MSE = variance + bias squared, combines the two. A biased estimator with a small variance can have a smaller MSE than an unbiased one with a large variance, which is why biased estimators such as the ratio estimator are widely used. Survey design aims to minimise total error for a given cost, not sampling error alone.
In probability sampling every unit of the population has a known, non-zero chance of selection, and the sample is chosen by a random mechanism that follows those chances. This is what allows unbiased estimation, the calculation of standard errors from the sample and the statement of confidence intervals. Simple random, stratified, systematic, cluster, multistage and probability-proportional-to-size sampling are all probability methods. In non-probability sampling (purposive or judgement selection, convenience samples, quota samples, volunteers) the selection probabilities are unknown. The sample may happen to be good, but there is no objective way to measure its error, and selection bias is common: an enumerator choosing 'typical' fields tends to choose accessible, well-managed ones. Early crop-estimation debates were partly about this; P. C. Mahalanobis in India and the later work of P. V. Sukhatme established random selection of fields and plots as the basis of crop-cutting surveys.
A frame is the list, map or other device from which the sample is drawn. A list frame enumerates the units: holdings from the agricultural census, villages from the population census, members of a cooperative. An area frame divides the land into segments with identifiable boundaries, and segments are sampled and visited; area frames are widely used for crop area and yield surveys because land does not move and does not depend on administrative lists. Frames have four common defects: omissions (units missing from the list), duplicates (a unit appearing twice, which raises its chance of selection), out-of-scope units (holdings that no longer farm), and outdated or wrong auxiliary information. Omissions cause undercoverage bias that no estimator can correct; duplicates and out-of-scope units can be handled if they are detected in the field. Keeping frames current is a major part of the work of a statistical office.
Two probabilities must be distinguished. The selection probability at a draw is the chance that a unit is selected at one particular draw. The inclusion probability, written pi_i, is the chance that unit i ends up anywhere in the sample. In simple random sampling of n units out of N, each unit has selection probability 1/N at the first draw, and its inclusion probability is n/N, whether the sampling is done by successive draws without replacement or by choosing one of the equally likely samples. The second-order inclusion probability pi_ij is the chance that both units i and j are in the sample; for SRSWOR it is n(n - 1)/[N(N - 1)]. Inclusion probabilities determine the weights. A unit included with probability pi_i represents 1/pi_i population units, so the Horvitz-Thompson estimator of the total, the sum over the sample of y_i / pi_i, is unbiased for any design with known positive pi_i. For simple random sampling the weight is N/n and the estimator reduces to N times the sample mean. The theory of unequal-probability designs built on this idea is in the Sampling Theory II subject.
Random samples are drawn with random numbers from a table or a computer generator. With N = 850 units numbered 1 to 850, three-digit random numbers are read; numbers from 001 to 850 select units and the rest are rejected. To waste fewer numbers, the remainder method divides each random number by N and uses the remainder (with 0 taken as N), using only random numbers below the largest multiple of N that fits in the digits read, so that every unit keeps the same chance. In sampling without replacement, a number that repeats a unit already selected is skipped. The lottery method, drawing numbered slips from a well-mixed container, is equivalent for small populations.
Simple random sampling (SRS) selects n units from N so that every possible sample of size n has the same chance. Without replacement (SRSWOR) there are C(N, n) = N! / [n!(N - n)!] possible samples, each with probability 1 / C(N, n). With replacement (SRSWR) a unit is returned after each draw and can be selected again; there are N to the power n ordered samples, each equally likely. Write Y-bar for the population mean, S squared = sum of (Y_i - Y-bar) squared / (N - 1) for the population variance with divisor N - 1, and sigma squared = sum of (Y_i - Y-bar) squared / N, so that sigma squared = (N - 1) S squared / N. The sample mean is y-bar and the sample variance s squared = sum of (y_i - y-bar) squared / (n - 1). The sampling fraction is f = n / N and 1 - f is the finite population correction (fpc).
In SRSWOR the sample mean y-bar is unbiased for Y-bar, and its variance is V(y-bar) = (1 - f) S squared / n = [(N - n) / N] S squared / n. The sample variance s squared is unbiased for S squared, so V(y-bar) is estimated unbiasedly by (1 - f) s squared / n. The population total Y = N Y-bar is estimated by N y-bar, with variance N squared (1 - f) S squared / n. The finite population correction shows that sampling a large fraction of a small population reduces the variance: if n = N the variance is zero, as it should be. When f is small, below about 5%, the fpc is close to one and is often ignored. The important practical point is that precision depends mainly on n, not on the fraction sampled: a sample of 400 from a population of 40,000 is nearly as precise as a sample of 400 from 4,000,000 when the variability per unit is the same.
In SRSWR the sample mean is also unbiased, with variance V(y-bar) = sigma squared / n = (N - 1) S squared / (N n). The sample variance s squared is unbiased for sigma squared here. Comparing the two schemes, V_WR / V_WOR = (N - 1) / (N - n), which is greater than 1 whenever n > 1: sampling without replacement is always at least as precise, because a repeated unit adds no new information. SRSWR is easier to analyse and is the model behind several unequal-probability schemes, and for large N the two give practically the same results. In SRSWR the number of distinct units in the sample is itself random. The mean of the distinct units is also unbiased and has a smaller variance than the mean of all n draws, which is another way of seeing why repeats are wasteful.
For reasonably large samples the sampling distribution of y-bar is close to normal (by the central limit theorem for finite populations), and an approximate 95% confidence interval for the mean is y-bar plus or minus 1.96 times the estimated standard error, the square root of (1 - f) s squared / n. For small samples from populations that are not too skewed, t with n - 1 degrees of freedom replaces 1.96. Highly skewed populations, common for farm sizes, livestock numbers and incomes, need larger samples before the normal approximation is good; stratifying the largest units out, or a census of them, is the usual remedy.
A proportion is the mean of a variable that equals 1 if a unit has an attribute and 0 otherwise: the proportion of holdings using drip irrigation, of farmers insured, of fields infested. If P is the population proportion and Q = 1 - P, then for the 0-1 variable S squared = N P Q / (N - 1). The sample proportion p is unbiased for P in SRS. Under SRSWOR its variance is V(p) = [(N - n) / (N - 1)] P Q / n, and an unbiased estimate is v(p) = (1 - f) p q / (n - 1). Under SRSWR, V(p) = P Q / n, estimated by p q / (n - 1). The variance P Q / n is largest when P = 0.5, which is why sample-size calculations for proportions without prior knowledge assume P = 0.5. Relative precision is poor for rare attributes: estimating P = 0.02 with a coefficient of variation of 10% needs a very large sample, because the CV of p is the square root of Q / (n P).
To estimate a mean so that the error is within plus or minus d with confidence 1 - alpha, the required sample size ignoring the fpc is n0 = z squared S squared / d squared, where z is the standard normal value (1.96 for 95%). Including the fpc, n = n0 / (1 + n0 / N), which matters only when n0 is a noticeable fraction of N. If the precision is stated in relative terms, an error of at most r (for example 5%) of the mean, then n0 = z squared C squared / r squared, where C = S / Y-bar is the population coefficient of variation. S or C is not known before the survey, so it is taken from a previous survey of the same population, a pilot survey, or a two-stage procedure in which a first sample estimates S. Because precision is proportional to the square root of n, halving the margin of error requires four times the sample.
For a proportion with absolute margin d, n0 = z squared P Q / d squared, with n = n0 / (1 + (n0 - 1) / N) or, as an approximation, n0 / (1 + n0 / N). With 95% confidence, d = 0.05 and P unknown (taken as 0.5), n0 = 1.96 squared x 0.25 / 0.0025, about 384, the familiar figure. These formulas assume simple random sampling. Real surveys are usually clustered and multistage, so the result is multiplied by a design effect (the ratio of the actual variance to the SRS variance) taken from earlier surveys, and then divided by the expected response rate. A survey that must give estimates for several domains (districts, farm-size classes) needs the calculation for each domain, which often drives the total sample size up far more than the overall precision does. Finally, the budget sets an upper limit: the sample size is a compromise between precision and cost, and the cost model in the chapter on stratified sampling shows how to optimise it.
In stratified sampling the population of N units is divided into L non-overlapping groups, the strata, of sizes N_1 to N_L, and an independent sample is drawn from each, usually by SRS. Stratification is used for four reasons: to gain precision when strata are internally homogeneous; to obtain estimates of known precision for each stratum (each district or farm-size class) as well as for the whole; for administrative convenience, since field offices are often organised by region; and to use different sampling methods where problems differ, for example a complete enumeration of the few very large farms and a sample of the many small ones. In agricultural surveys strata are commonly formed by geography (districts, agro-climatic zones), by size (area operated, herd size), or by irrigation status, using information from the census or the previous survey.
Let W_h = N_h / N be the stratum weight, y-bar_h and s_h squared the sample mean and variance in stratum h, n_h the stratum sample size and f_h = n_h / N_h. The stratified estimator of the population mean is y-bar_st = sum of W_h y-bar_h. It is unbiased, and with SRSWOR in each stratum its variance is V(y-bar_st) = sum of W_h squared (1 - f_h) S_h squared / n_h, estimated by replacing S_h squared with s_h squared. The total is estimated by N y-bar_st = sum of N_h y-bar_h. The variance contains only the within-stratum variances S_h squared. Variation between stratum means does not enter, which is why stratification pays when strata differ much in their means and are homogeneous inside. Note that y-bar_st is not in general equal to the overall sample mean; they coincide only when the sample is self-weighting, as with proportional allocation.
Given a total sample n, it can be shared among strata in several ways. Equal allocation gives each stratum n/L, useful when stratum estimates of similar precision are wanted. Proportional allocation sets n_h = n N_h / N = n W_h, giving a self-weighting sample; its variance is V_prop = [(1 - f)/n] sum of W_h S_h squared. Neyman (optimum for fixed n) allocation sets n_h = n N_h S_h / sum of N_h S_h, which minimises V(y-bar_st) for fixed total n; its variance is V_opt = (sum of W_h S_h) squared / n - (sum of W_h S_h squared) / N. Optimum allocation with costs, when enumerating a unit in stratum h costs c_h and total cost is C = c_0 + sum of c_h n_h, sets n_h proportional to N_h S_h / square root of c_h: sample more in a stratum that is larger, more variable or cheaper. If the allocation formula gives n_h > N_h for a stratum, that stratum is enumerated completely and the rest of the sample is reallocated among the others. Neyman allocation needs the S_h, which are estimated from past data; moderate errors in them cost little precision.
Ignoring terms in 1/N_h, the variances satisfy V_opt <= V_prop <= V_ran, where V_ran is the variance of the mean of an SRS of the same total size. The gain of proportional allocation over SRS is approximately [(1 - f)/n] sum of W_h (Y-bar_h - Y-bar) squared, the variation between stratum means; the further gain of Neyman over proportional allocation is approximately (1/n) sum of W_h (S_h - S-bar) squared, where S-bar = sum of W_h S_h, which is large only when the stratum standard deviations differ greatly. In farm surveys, where variability rises steeply with farm size, Neyman allocation to the large-farm strata often gives a large gain. Post-stratification, classifying the units of an SRS into strata after selection and weighting by known W_h, recovers most of the gain of proportional allocation when the sample is large, and is used when the stratum of a unit is not known in advance.
The best stratification variable would be the character being estimated itself, but that is unknown before the survey. In practice a closely correlated auxiliary variable known for every unit is used: the area under the crop in the previous census for a survey of crop production, the herd size in the livestock census for milk production, the geographical zone for agro-climatic variation. The more closely the stratification variable is correlated with the study variable, the closer the gain comes to what stratifying on the study variable would give. Surveys with many study variables compromise, often using geography and a size measure together.
When strata are formed by cutting a continuous size variable into classes, the boundaries matter. For Neyman allocation, Dalenius showed that optimum boundaries satisfy equations that are hard to solve directly. The cumulative square-root-of-frequency rule of Dalenius and Hodges gives a good approximation: form a fine frequency distribution of the stratification variable, take the square root of each class frequency, cumulate these square roots, and choose boundaries that divide the cumulative total into L equal parts. Other approximations equalise W_h S_h across strata, or (for proportional allocation) the products W_h times the stratum range. A frequently cited conclusion of Cochran is that, with a good stratification variable, the gain in precision rises quickly over the first few strata and then levels off, so beyond about six strata further subdivision of the same variable adds little; it is usually better to stratify by an additional variable or to spend resources on a larger sample. Very large units, whose values dominate totals, are best placed in a completely enumerated top stratum.
Fine stratification can be pushed to the point of one unit selected per stratum, which gives the most precise estimator but leaves no within-stratum variance estimate, since s_h squared needs at least two units. The usual remedy is collapsing: strata are grouped in pairs (or larger groups) that are similar, and the variance is estimated from the differences between the paired units, treating each pair as if it were one stratum with two units. With two strata of similar size collapsed, the variance contribution of the pair is estimated from (Y-hat_1 - Y-hat_2) squared, where the Y-hat are the estimated stratum totals; weighted versions exist for strata of unequal size. This collapsed-strata variance estimator is biased upward, because it includes part of the variation between the strata that were combined; the bias is small when the paired strata have similar means. It is therefore conservative: confidence intervals are somewhat wider than they need to be. The same device is used in multistage surveys where one primary unit is selected per stratum, as in many national household and agricultural surveys.
In linear systematic sampling with N = nk units arranged in order, a random number r between 1 and k is chosen and the units r, r + k, r + 2k, ..., r + (n - 1)k form the sample; k is the sampling interval. Only k distinct samples are possible, each with probability 1/k, and each unit has inclusion probability 1/k = n/N, so the sample mean is unbiased for the population mean. Systematic sampling is easy to carry out in the field, for example selecting every tenth household along a listing, every twentieth plant along a row, or points at fixed spacing on a grid, and it spreads the sample evenly over the population. When N is not a multiple of n, the linear method gives samples of unequal size and the sample mean becomes slightly biased. Circular systematic sampling, attributed to Lahiri, avoids this: the units are arranged in a circle, k is taken as the integer nearest to N/n, a random start between 1 and N is chosen, and every kth unit is taken round the circle until n units are selected.
Arrange the N units in the k possible systematic samples (as columns). The variance of the systematic sample mean can be written V_sys = [(N - 1)/N] S squared - [k(n - 1)/N] S_wsy squared, where S_wsy squared is the variance among units within the same systematic sample. Systematic sampling is more precise than SRSWOR of the same size if and only if S_wsy squared > S squared, that is, if units within a systematic sample are more variable than the population as a whole. In terms of the intraclass correlation rho_w between pairs of units in the same systematic sample, V_sys = [S squared (N - 1) / (n N)] [1 + (n - 1) rho_w], so systematic sampling gains when rho_w is negative and loses when it is positive. Three standard cases follow. In a population in random order, systematic sampling behaves on average like SRS. With a linear trend, systematic sampling is better than SRS, but stratified sampling with one unit per stratum is better still, and end corrections (Yates) or balanced and centrally located systematic samples remove much of the trend effect. With periodic variation, such as a crop row pattern or a weekly cycle of market arrivals, systematic sampling is disastrous if k equals the period or a multiple of it, because every unit in the sample falls at the same phase; it is excellent if k is an odd multiple of half the period.
A single systematic sample, being in effect one cluster of n units chosen at random from k clusters, provides no unbiased estimator of its own variance. Practical approaches treat the sample as if it were random (acceptable for populations in random order), use successive differences between neighbouring units (suited to trends), or draw several systematic samples with independent random starts (interpenetrating sub-samples) and estimate the variance from the variation between their means. Systematic and grid sampling are used to select points in area frames, soil samples on a grid, plants along rows for pest counts, trees in forest inventory and households from village listings. The analyst should always ask whether the population could have a period matching the interval.
Often an auxiliary variable X, correlated with the study variable Y, is known for every unit or at least its population total or mean is known: last year's crop area or production, the census number of cattle, the geographical area of a village. If Y is roughly proportional to X, the ratio estimator uses this. With SRS of n units, the sample ratio is R-hat = y-bar / x-bar and the ratio estimator of the population mean is y-bar_R = R-hat X-bar = (y-bar / x-bar) X-bar; the total is estimated by R-hat X. Ratio estimation is also the natural way to estimate a population ratio itself, such as yield per hectare (total production over total area) or the proportion of area irrigated.
The ratio estimator is biased, because the expectation of a ratio is not the ratio of expectations. To first order, the bias of y-bar_R is [(1 - f)/n] (R S_x squared - S_yx) / X-bar, where R = Y-bar / X-bar and S_yx is the population covariance. The bias is of order 1/n and becomes negligible in large samples; a useful bound is that the ratio of the absolute bias to the standard error of R-hat is at most the coefficient of variation of x-bar, so the bias is unimportant when that CV is small, say below 0.1. To the same order, MSE(y-bar_R) = [(1 - f)/n] (S_y squared + R squared S_x squared - 2 R rho S_x S_y), where rho is the correlation between Y and X. It can be estimated by replacing population quantities with sample ones.
Comparing the MSE of y-bar_R with the variance (1 - f) S_y squared / n of the simple mean, the ratio estimator is more precise in large samples if and only if rho > (1/2)(C_x / C_y), where C_x and C_y are the coefficients of variation of X and Y. When X and Y have similar CVs, a correlation above 0.5 is enough. The ratio estimator is the best linear unbiased estimator when the relation between Y and X is a straight line through the origin and the variance of Y about the line is proportional to X, a model that often fits crop production against area. Because the bias can matter in small samples and in stratified surveys with many small strata, unbiased or almost unbiased ratio-type estimators have been developed. The Hartley-Ross estimator is exactly unbiased under SRS; others reduce the bias to a higher order by jackknifing or by sampling with probability proportional to x-bar (Lahiri-Midzuno-Sen scheme), under which the ordinary ratio estimator becomes unbiased.
In stratified sampling there are two ratio estimators. The separate ratio estimator forms a ratio estimate in each stratum and adds them: the sum of (y-bar_h / x-bar_h) X_h. The combined ratio estimator forms one ratio from the stratified means: (y-bar_st / x-bar_st) X. The separate estimator is more efficient if the ratios differ between strata, but its bias accumulates across strata and can be serious when stratum samples are small; the combined estimator is safer then and needs only the overall total of X, not the stratum totals.
When the relation between Y and X is linear but does not pass through the origin, the regression estimator is better than the ratio estimator. With SRS and known X-bar, the linear regression estimator of the population mean is y-bar_lr = y-bar + b (X-bar - x-bar), where b is the sample regression coefficient of y on x, b = s_yx / s_x squared. The estimator adjusts the sample mean upward when the sample's x-bar is below the population mean X-bar and the slope is positive, and downward when x-bar is above it. If b were a fixed constant chosen in advance, the estimator would be unbiased (this is the difference estimator of the next chapter). With b estimated from the sample, the regression estimator has a bias of order 1/n, which is usually small.
To first order, V(y-bar_lr) = [(1 - f)/n] S_y squared (1 - rho squared). The factor (1 - rho squared) shows the gain over the simple mean: with rho = 0.8 the variance is 36% of that of the simple mean, and with rho = 0.9 it is 19%. The variance is estimated by [(1 - f)/n] [1/(n - 2)] times the sum of squared residuals about the fitted line. In large samples the regression estimator is at least as precise as both the simple mean and the ratio estimator. Its variance equals that of the ratio estimator only when the population regression coefficient beta equals R = Y-bar / X-bar, that is, when the population regression line passes through the origin. When the line passes close to the origin, the ratio estimator is nearly as good and simpler; when it has a large intercept, the regression estimator can be much better. In small samples and with outliers, the estimated slope can be unstable, and the large-sample comparison may not hold.
In stratified sampling there are separate and combined regression estimators, with the same trade-off between efficiency and small-sample bias as for ratio estimators. Regression estimation with several auxiliary variables leads to the generalised regression (GREG) and calibration estimators used by statistical offices, in which sample weights are adjusted so that weighted sample totals of the auxiliaries match known population totals; these are covered in the Small Area Estimation subject. In crop statistics, regression estimators are used to combine expensive and cheap measurements: for example, the eye estimate of yield or a satellite index available for all fields (X) with crop-cutting yields on a sample (Y), or the patwari's area record (X) with a sample of measured areas (Y). Double sampling, where X itself is observed on a large first-phase sample because it is not known for all units, extends the same ideas.
The difference estimator uses a constant k fixed in advance, not estimated from the sample: y-bar_D = y-bar + k (X-bar - x-bar). Because k is fixed and x-bar is unbiased for X-bar, y-bar_D is exactly unbiased for Y-bar under SRS, whatever the population. Its variance is V(y-bar_D) = [(1 - f)/n] (S_y squared + k squared S_x squared - 2 k S_yx). The variance is minimised when k = beta = S_yx / S_x squared, the population regression coefficient, and the minimum equals the large-sample variance of the regression estimator, [(1 - f)/n] S_y squared (1 - rho squared). The difference estimator is useful when a good value of k is known from earlier surveys. When k = 1 it simply adjusts y-bar by the difference between the population and sample means of X, which suits situations where Y and X measure the same thing at two times (this year's and last year's area); it beats the simple mean when 2 rho S_y > S_x, i.e. rho > S_x / (2 S_y).
When Y and X are negatively correlated, the ratio estimator is worse than the simple mean. Murthy's product estimator, y-bar_P = y-bar x-bar / X-bar, is then the analogue: it scales y-bar up when x-bar is above its population mean. It is biased, with first-order bias [(1 - f)/n] S_yx / X-bar, and its first-order MSE is [(1 - f)/n] (S_y squared + R squared S_x squared + 2 R rho S_x S_y). Since rho is negative, the last term reduces the variance. The product estimator is more precise than the simple mean if rho < -(1/2)(C_x / C_y). Examples of negative correlation in agricultural surveys are pest damage against yield, the proportion of area under a crop failing against rainfall, or the price of a commodity against arrivals in a market.
The choice rests on the relation between Y and X. With a strong positive relation through the origin and variance rising with X, use the ratio estimator. With a strong linear relation and a clear intercept, use the regression estimator, or the difference estimator if a reliable slope is known in advance. With a strong negative relation, use the product estimator or regression. With weak correlation, roughly |rho| below C_x / (2 C_y), the simple mean is as good as any of them and avoids bias. Ratio, product and regression estimators are all of the general form y-bar + (something) (X-bar - x-bar); they differ in how the adjustment coefficient is chosen. In all cases the auxiliary population mean or total must be accurately known, since any error in it passes directly into the estimate.
Students meet this subject as derivations and numericals in examinations and as the backbone of any survey-based thesis. They must be able to derive V(y-bar) under SRSWOR, show why Neyman allocation minimises variance, state when a ratio estimator beats the mean, and compute a sample size from a CV. For a thesis that uses farm-household or field surveys, they design the sample, compute weights and standard errors, and defend the sample size before the advisory committee.
A survey statistician five years in designs and runs samples: allocating crop-cutting experiments across districts in proportion to area or by Neyman allocation using last season's variances, choosing the stratification variable from the census, deciding whether to use the patwari area or a satellite index as an auxiliary for a ratio or regression estimator, computing the sample size a new livestock survey needs for district estimates, and checking whether a systematic selection along village lists risks periodicity. They also explain to administrators why a district estimate from a state sample has a wide margin.
A senior head of a survey programme decides the architecture of a country's or state's agricultural surveys: the frame (list or area), the strata, the allocation rule, the estimators and the variance-estimation method, balancing precision for state and district estimates against budget and field capacity. They approve changes when new auxiliary data such as satellite classifications become available and set the documentation and quality standards so that published standard errors are trustworthy.
A PhD with long experience develops new estimators and designs, for example ratio-type estimators with smaller bias, optimal stratification algorithms or model-assisted estimators using remote-sensing auxiliaries, and teaches the theory. They advise statistical offices on methodology, review survey designs and judge, from the theory in this subject, whether a proposed estimator is likely to improve on what is in use.
This manual covers equal-probability designs and the basic auxiliary-variable estimators. PPS sampling and the Horvitz-Thompson theory in depth, cluster and multistage sampling, double sampling, successive occasions, randomised response, non-response and imputation are in Sampling Theory II; small-area estimation and calibration in their own subject. Formulas are given to first order (large-sample approximations) and are stated without full proofs; exact small-sample results for ratio and regression estimators are not covered. Statistical tables are not reproduced; questions state the values they need.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions. Every numerical answer was computed with a script; the values needed are stated in the question.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.