Agricultural Statistics & Data Science · Time series, forecasting and modelling · also Crop Yield Forecasting, Growth Models, Statistical Modelling in Agriculture
Forecasting harvests before they happen: Fisher's regression integral and weather-indices models, pre-harvest forecasts from biometric characters and farmers' appraisal, agro-meteorological and remote-sensing models, non-linear growth curves (logistic, Gompertz, Richards), and how statistical and crop-simulation models are combined.
In the exams: AICE-JRF Agricultural Meteorology 'statistical models' (present); ICAR MSc/PhD core in modelling; national crop-forecasting programmes (applied); light in national exams
A crop forecast is an estimate of area, yield or production made before harvest. Governments need it to plan procurement, buffer stocks, imports and exports, and food distribution; traders and processors need it to plan purchases; insurers and relief agencies need it to anticipate losses. The final estimate of production comes after harvest from area enumeration and crop-cutting experiments, the methods of the Crop Estimation Surveys subject of this track. A forecast must come earlier, while the crop is still standing, and must therefore rely on what can be observed in season: the weather so far, the condition of plants, farmers' expectations and what satellites see.
This subject covers the statistical models used for that purpose. It begins with the purpose and history of pre-harvest forecasting, then treats models built on plant biometric characters, on farmers' eye estimates, and on weather through Fisher's regression integral and the weather-indices approach developed at ICAR-IASRI. It moves to agro-meteorological and composite models, remote-sensing models with vegetation indices, and then to non-linear growth models, logistic, Gompertz, monomolecular and Richards, and the iterative methods used to fit them. The later chapters cover crop simulation models and their statistical calibration, machine-learning and hybrid models, and the evaluation of forecasts and their use in policy.
In examinations the subject appears as statistical models in agricultural meteorology, as crop-yield forecasting in agricultural statistics, and as growth-curve fitting in modelling courses. National entrance papers ask about it lightly; master's and PhD programmes and the scientists of forecasting agencies use it heavily. The regression theory behind most of these models is in Linear Models and Regression, and the time-series methods for trend and forecast evaluation are in Time Series Analysis.
Final production estimates come months after harvest, too late for many decisions. A government deciding whether to allow exports of rice, to release wheat from public stocks or to procure pulses at a support price needs an estimate while the crop is still in the field. Forecasts are made at several stages: at sowing (area and intentions), in mid-season (crop condition and weather so far) and close to harvest (yield components and objective measurements). Each forecast is revised as information arrives, and accuracy improves as harvest approaches. The value of a forecast lies in its timeliness as much as its accuracy: a forecast two months before harvest with a 10 per cent error may be more useful than one a week before harvest with a 3 per cent error, because decisions on trade and stocks take time. Forecasts may be of area, of yield per unit area, or of production as area times yield; area is usually estimated more reliably than yield, and most statistical forecasting effort goes into yield.
Formal crop reporting began with subjective reports: in many countries village officials or crop reporters assessed the condition of crops as a proportion of a normal crop. In India the traditional anna (annewari) system expressed the expected crop in annas of a rupee, 16 annas meaning a normal crop, and the system survives in some states for declaring scarcity. Statisticians soon sought objective methods. R. A. Fisher's 1924 study of rainfall and wheat yields at Rothamsted introduced the idea of a regression integral relating yield to the distribution of rainfall through the season. In the United States, the Department of Agriculture's crop reporting service, later the National Agricultural Statistics Service (NASS), combined farmer surveys with objective yield surveys in which plant counts and fruit measurements are taken in sample plots. In India, the Indian Agricultural Statistics Research Institute (IASRI) developed models based on biometric characters, farmers' appraisal and weather indices from the 1950s onwards.
Since the 1980s, remote sensing has added an independent view of crop area and condition. India's Crop Acreage and Production Estimation (CAPE) project used satellite data with ground truth for major crops, and the FASAL programme (Forecasting Agricultural output using Space, Agro-meteorology and Land-based observations) combined satellite, weather and survey information to give multiple in-season forecasts; the Mahalanobis National Crop Forecast Centre was set up under the agriculture ministry to run such forecasts. Internationally, the United States publishes monthly supply and demand estimates, the European Union's MARS (Monitoring Agricultural Resources) activity at the Joint Research Centre publishes crop-monitoring bulletins using agro-meteorological models and remote sensing, FAO's Global Information and Early Warning System (GIEWS) monitors food-insecure countries, and the GEOGLAM initiative and the Agricultural Market Information System (AMIS) coordinate crop monitoring for major traded crops. Details of these programmes change, and current documentation should be consulted for their present methods.
A forecasting model must use only information available at the time of forecast; a model that uses rainfall up to harvest cannot forecast two months earlier. It must be validated on years not used to fit it, because agricultural series are short (often 20 to 40 years) and it is easy to overfit with many weather variables. It should be stable when refitted with a new year added, and its errors should be quantified so that users know how much to trust it. It must also be operational: data must arrive on time, at the spatial level (district, state, country) at which the forecast is needed. These criteria are the thread that runs through every model in this subject.
Biometric characters are measurable features of plants that relate to final yield: plant population per unit area, number of tillers or branches per plant, plant height, number of ear-heads, panicles, bolls or pods per plant, length of ear-head, and later the number of grains. They are observed in sample plots within farmers' fields during the season, usually at stages such as maximum tillering, flowering and grain formation. The sampling design follows that of crop-cutting surveys: a multistage design with districts or blocks, villages, fields and plots, so that estimates can be aggregated with known precision. Because the same fields are later harvested in crop-cutting experiments, the relationship between in-season characters and final yield can be estimated from the same units. The method was developed and tested by IASRI for crops including wheat, rice, sugarcane, cotton and jute.
The basic model regresses observed plot yield on the biometric characters measured earlier in the season: yield = b0 + b1 x1 + ... + bp xp + error. The model is fitted on one season's data and used to forecast the next, or fitted on pooled data from several seasons. Characters measured closer to harvest are more strongly related to yield but give a later forecast, so models are fitted stage by stage. Because characters are strongly correlated with one another (plant population and number of ear-heads, for example), the regression may suffer from multicollinearity; selection of characters by stepwise methods, principal component regression (regressing yield on the first few principal components of the characters) and ridge regression have all been used. Damage due to pests, diseases or lodging, recorded as a proportion of plants affected, is added as a further variable, since a crop with good early growth can still fail.
An alternative uses growth indices: the characters at successive stages are combined into indices weighted by their correlation with yield, giving a single summary of crop growth to date, analogous to the weather indices of Chapter 4. Models based on biometric characters give good accuracy close to harvest and work at the level of the field and the sampling domain. Their limits are cost and timeliness. Taking plant counts and measurements in thousands of plots requires trained staff, and data must be collected, transmitted and processed quickly for the forecast to be useful. For this reason operational systems now combine a smaller ground sample with remote sensing and weather data, using the biometric measurements to calibrate or validate the other sources.
A farmer who has grown a crop on the same land for years can judge, while the crop stands, what it will yield. An eye estimate or farmer's appraisal records that judgement, usually as expected yield per unit area, sometimes as a proportion of a normal crop. Crop reporters, village officers and extension workers also make eye estimates. Such estimates are cheap, fast and can be collected over a very wide area, which makes them attractive for early forecasts. Their weakness is subjectivity: farmers may understate expected yields to protect claims for relief or loan waivers, or overstate them out of optimism; officials may be influenced by administrative pressure; and judgements tend to regress towards a normal crop, understating both very good and very poor years.
The statistical approach does not take eye estimates at face value but calibrates them against objective yields. In a sample of fields, both the farmer's appraisal and the crop-cutting yield are recorded. A regression of observed yield on appraised yield, fitted on these fields, is then applied to appraisals from a much larger sample, giving a regression (or ratio) estimator in the sense of double sampling: the large first-phase sample gives cheap appraisals, and the small second-phase sample gives the relationship between appraisal and true yield. The sampling theory is that of double sampling for regression and ratio estimation, covered in Crop Estimation Surveys. The gain in precision depends on the correlation between appraised and observed yields; IASRI studies found useful correlations in several crops, but the correlation varies by crop, region and season and must be re-estimated rather than assumed.
Appraisal-based forecasts are useful for early estimates, for crops and regions without enough crop-cutting experiments, and as one input to composite forecasts. Several cautions apply. Appraisals must be collected before the farmer knows the final yield, or the forecast becomes a disguised final estimate. Incentives must be considered: when payouts from relief or insurance depend on reported losses, appraisals may be biased, and the bias may change from year to year with the policy. Qualitative crop-condition ratings (for example, very poor to excellent, as used in some national crop-progress reports) can also be converted into yield forecasts by regressing past yields on the proportion of the crop in each condition class; the same calibration logic applies.
Yield depends not only on the total rainfall in a season but on when it falls. Fisher (1924) treated the effect of rainfall on yield as a continuous function of time through the season: yield = A0 + integral over the season of a(t) w(t) dt, where w(t) is the rainfall at time t and a(t) is the regression coefficient of yield on rainfall at that time. To make the problem estimable with few years of data, he assumed that both the coefficient function a(t) and the rainfall w(t) could be expressed as polynomials in time using orthogonal polynomials. The integral then reduces to a linear regression of yield on a small number of constants derived from the rainfall polynomial, and the coefficient curve a(t) can be reconstructed to show at which stages rainfall helps or harms yield. The approach is elegant and makes efficient use of short series, but with real weather data the polynomial assumptions are restrictive, and fitting requires care.
Hendricks and Scholl (1943) simplified Fisher's approach for weekly data. They assumed that the effect of a weather variable in week w could be expressed as a second-degree polynomial in w, so that the yield model became a regression on three derived variables: the sum of weekly weather values, the sum of week number times weather, and the sum of week number squared times weather. They also allowed for the joint (interaction) effect of two weather variables in a similar way. This reduced the number of parameters to estimate from one per week to a few per variable, an important saving when only 20 to 30 years of yield data are available. The weekly framework, with standard meteorological weeks, became the basis for later work.
IASRI workers developed a more flexible approach in which weekly weather is combined into indices weighted by correlation with yield. For a weather variable i observed in weeks w = 1, ..., m of the forecast period, the unweighted index is Z(i,0) = sum over w of x(i,w), and the weighted index is Z(i,1) = sum over w of r(i,w) x(i,w), where r(i,w) is the correlation coefficient between yield (usually detrended or adjusted for trend) and the weather variable in week w, computed over past years. Interaction indices for two variables use the product x(i,w) x(i',w) with weights from the correlation of yield with that product. The forecast model regresses yield on a time trend and the selected indices, usually chosen by stepwise regression. Using correlations as weights lets the model emphasise weeks that matter, critical stages such as flowering, without estimating a separate coefficient for each week. Models are fitted at district level with weather up to about two months before harvest and tested on later years.
Because each index is built from correlations computed on the same years used to fit the regression, the in-sample fit is optimistic; honest evaluation needs forecasts for years left out (Chapter 11). The trend term carries technology change (new varieties, irrigation, fertiliser), and its form matters: a linear trend fitted over a period that includes a technological jump will distort the weather effects. Weather station data may not represent the district, and missing observations must be handled. With many weather variables and weeks, the number of candidate indices grows, and stepwise selection on short series is prone to chance findings. Models therefore need to be re-examined as new years are added and judged by their performance on independent years, not by R-squared.
Agro-meteorological models replace raw weather with variables that describe how the weather acts on the crop. Growing degree-days (thermal time) accumulate the daily mean temperature above a base temperature, (Tmax + Tmin)/2 - Tbase, summed over days (negative values set to zero), and track crop development better than calendar days; base temperatures differ by crop and are taken from crop-specific literature. Water-balance models track soil moisture from rainfall, irrigation and evapotranspiration and compute the extent to which the crop's water need has been met. FAO's Water Requirement Satisfaction Index (WRSI) expresses the ratio of actual to potential crop evapotranspiration accumulated over the season as a percentage, and is widely used for rainfed crops in food-security monitoring. Indices of heat stress (days above a critical temperature at flowering), frost, and drought (such as the standardised precipitation index) capture extreme events that average weather misses.
The simplest agro-meteorological model regresses yield on a trend and one or more such indices, for example WRSI at the end of the season, cumulative degree-days in grain filling, or the number of hot days at anthesis. Because these variables carry crop physiology, they often generalise better across years and regions than raw weekly weather, and they need fewer parameters. Stage-wise models compute the variables separately for vegetative, reproductive and grain-filling phases, defined by phenology rather than calendar weeks, so that a late-sown crop is not compared with an early-sown one week by week. The phenology itself may be estimated from thermal time or from remote sensing (Chapter 6).
No single source of information is best at every stage of the season. Composite models combine variables from different sources: weather indices, agro-meteorological indices, biometric characters, farmers' appraisal, crop-condition ratings and remote-sensing indices. One approach enters them together in a regression and selects by stepwise or penalised methods; another builds separate forecasts from each source and combines them with weights based on their past accuracy, the forecast-combination idea of Time Series Analysis. Combining tends to reduce error when the sources carry different information and their errors are not highly correlated. The cost is complexity: each source must arrive on time, and the model must be re-examined when one source changes, for example when a new satellite replaces an old one.
Green leaves absorb strongly in the red part of the spectrum, where chlorophyll uses light for photosynthesis, and reflect strongly in the near-infrared (NIR), because of the internal structure of leaves. Bare soil and water lack this contrast. Vegetation indices exploit it. The Normalised Difference Vegetation Index is NDVI = (NIR - Red)/(NIR + Red), which ranges from -1 to +1; water gives negative values, bare soil values near zero, and dense green crops high positive values. NDVI rises through the vegetative phase, peaks around flowering and falls during senescence, tracing the crop's growth curve. Other indices address NDVI's weaknesses: NDVI saturates at high leaf area index (LAI), so dense canopies look alike; the Enhanced Vegetation Index (EVI) reduces saturation and atmospheric effects; the Soil-Adjusted Vegetation Index (SAVI) reduces the influence of soil background in sparse canopies.
Before yield can be forecast, the crop must be found and its area measured. Satellite images are classified into land-cover classes by supervised methods (maximum likelihood, random forest, support-vector machines) trained on ground-truth points collected in the field. Multi-date images help separate crops by their different growth calendars. Optical sensors cannot see through cloud, which is a serious limit for monsoon (kharif) crops; synthetic aperture radar (SAR) can, and SAR backscatter is used to map rice in particular, since flooded fields at transplanting have a distinctive signal. Classification errors must be estimated from an accuracy assessment (a confusion matrix with independent reference data), and area estimates are corrected for misclassification, or combined with ground survey data by regression estimators, rather than taken from pixel counts alone.
Spectral yield models relate yield to a vegetation index. The simplest regress district yield on the peak NDVI of the season, or on the NDVI integrated over the reproductive period, together with a trend. Others estimate LAI or the fraction of absorbed photosynthetically active radiation (fAPAR) from the index and feed it into a light-use-efficiency model, in which biomass equals the sum over days of incident radiation times fAPAR times a light-use efficiency, and yield equals biomass times a harvest index. Satellite estimates can also be assimilated into crop simulation models (Chapter 9). Spectral models are spatially complete and available every few days from sensors such as MODIS, Landsat, Sentinel-2 and the Indian Resourcesat series, but they have limits: indices measure greenness, not grain; a crop can be green and still fail at flowering from heat; mixed pixels in fragmented smallholdings blur the signal; and relationships change between sensors, so long time series need cross-calibration.
Crop dry matter, plant height, leaf area, fruit size, animal weight and the spread of a disease all grow along S-shaped (sigmoid) or saturating curves: slow at first, faster in the middle, and levelling off at a maximum. Polynomials can approximate such curves over the observed range, but their parameters have no biological meaning and they behave badly when extrapolated. Mechanistic non-linear models have parameters that mean something: an asymptote (the final size), a rate, and a parameter that fixes the position of the curve in time. A model is non-linear in the statistical sense when it is non-linear in its parameters and cannot be made linear by a transformation of the response alone. In the formulas below, Y(t) is size at time t, K the asymptote, r the rate and b a location constant fixed by the initial size.
The logistic model arises from the differential equation dY/dt = rY(1 - Y/K): growth is proportional to current size and to the remaining room to grow. Its solution is Y(t) = K/(1 + b exp(-rt)). It is symmetric about its inflection point, which occurs at Y = K/2, at time t = ln(b)/r, where the growth rate is greatest at rK/4. The relative growth rate (1/Y) dY/dt = r(1 - Y/K) falls linearly with size. The logistic can be linearised when K is known: ln(K/Y - 1) = ln b - rt. The Gompertz model, Y(t) = K exp(-b exp(-rt)), arises when the relative growth rate falls exponentially with time. It is asymmetric: its inflection occurs at Y = K/e, about 37 per cent of the asymptote, at t = ln(b)/r, so it rises faster early and approaches the asymptote slowly. Gompertz curves often describe animal growth and some tumour and population growth; logistic curves describe many plant and population processes.
The monomolecular (Mitscherlich or Brody) model, Y(t) = K(1 - b exp(-rt)), arises when growth is proportional to the remaining room only, dY/dt = r(K - Y). It has no inflection point: growth is fastest at the start and slows steadily, a concave curve suitable for responses such as yield to fertiliser (Mitscherlich's law of diminishing returns) and for growth that starts after an initial phase has passed. The Richards model adds a shape parameter m that controls where the inflection occurs. One common form is Y(t) = K(1 + b exp(-rt)) to the power (-1/m). With m = 1 it is the logistic; as m tends to zero it approaches the Gompertz; with m = -1 and b negative it gives the monomolecular. The flexibility is useful, but the shape parameter is often poorly determined from noisy data and highly correlated with the other parameters, so the Richards model can be difficult to fit.
Model choice should start from biology and the shape of the data: is there an inflection, is it near the middle (logistic), early (Gompertz) or absent (monomolecular)? The asymptote K estimates final size; the time of inflection estimates when growth is fastest; the maximum absolute growth rate summarises vigour. Comparing treatments or varieties through these parameters, rather than through yields at a single date, uses all the growth data and asks biologically meaningful questions. When data cover only the early part of growth, the asymptote is poorly estimated, and extrapolation from any sigmoid model is unreliable. In time-series contexts, the same curves describe the adoption of a new technology or the growth of production, and they appear as trend models in Time Series Analysis.
For a model Y(i) = f(t(i), theta) + e(i), the least-squares estimate of the parameter vector theta minimises S(theta) = sum of (Y(i) - f(t(i), theta)) squared. Because f is non-linear in theta, the normal equations have no closed-form solution and must be solved iteratively. The Gauss-Newton method linearises f around the current estimate by a first-order Taylor expansion, f(theta) approximately f(theta0) + F(theta - theta0), where F is the matrix of partial derivatives of f with respect to the parameters, and solves the resulting linear least-squares problem for the step: delta = (F'F) inverse F'(Y - f(theta0)). The process repeats until the change in S or in theta is negligible. Gauss-Newton converges quickly near the solution but can diverge from poor starting values. The Levenberg-Marquardt method adds a multiple of the identity (or of the diagonal of F'F) to F'F, moving between a Gauss-Newton step and a steepest-descent step, and is the standard robust choice. Derivative-free methods such as Nelder-Mead are also used.
Iterative methods need starting values, and poor ones cause failure to converge or convergence to a local minimum. Good starting values come from the data and the meaning of the parameters: K a little above the largest observed value; the inflection time from the time at which the data reach about half the maximum (for the logistic); r from the slope of the linearised form, for example the regression of ln(K/Y - 1) on t for the logistic with K fixed at its starting value. Grid searches over plausible values and fitting the model first on transformed data are other strategies. Problems that signal a poorly posed fit include parameters that drift without bound, very large standard errors, and correlations between parameter estimates close to plus or minus 1; these usually mean the data do not contain enough information to estimate all parameters, for instance an asymptote with no data near the plateau, and the cure is a simpler model or more informative data, not a different algorithm.
Standard errors for non-linear least-squares estimates are based on the linear approximation at the solution: the estimated covariance matrix is s squared (F'F) inverse, with s squared = S(theta-hat)/(n - p). These are asymptotic and can be poor in small samples if the model is strongly curved; Bates and Watts distinguished intrinsic curvature (of the model itself) from parameter-effects curvature (due to the parameterisation), and re-parameterising can improve the approximation. Profile-likelihood intervals and bootstrap intervals are more reliable. For goodness of fit, the residual mean square and RMSE, plots of residuals against fitted values and time, and information criteria (AIC, BIC) for comparing models are standard. The R-squared from linear regression does not carry over cleanly, because there is generally no intercept-based decomposition of sums of squares; a pseudo R-squared (1 - residual SS/total corrected SS) is often reported but should not be the main criterion.
Growth data often violate the least-squares assumptions. Variance usually increases with size, which calls for weighted least squares or a transformation of both sides of the equation. Repeated measurements on the same plants or plots over time are correlated, which makes ordinary standard errors too small; residual autocorrelation can be checked with the Durbin-Watson statistic or a residual ACF and modelled with an autoregressive error structure. When curves are fitted to many plants or plots, non-linear mixed-effects models treat the parameters as varying randomly among units around population means, which handles both the correlation within units and the variation between them. These models link to the variance-component ideas of the Mixed Models subject of this track.
Crop simulation models compute crop growth day by day from weather, soil, management and the genetic characteristics of the cultivar. They represent processes: phenological development driven by temperature and day length, interception of light by the canopy, conversion to biomass, partitioning of biomass to leaves, stems, roots and grain, and the soil water and nitrogen balances that limit growth. Widely used models include the DSSAT family (CERES models for cereals and CROPGRO for legumes), APSIM, WOFOST (used in the EU's crop monitoring), FAO's AquaCrop (focused on water-limited yield), and InfoCrop developed in India. Because they encode physiology, simulation models can in principle handle conditions outside past experience, such as new sowing dates or climate change, which purely statistical models cannot.
A simulation model has many parameters. Genetic coefficients describe a cultivar: thermal time to flowering and maturity, photoperiod sensitivity, grain number and grain filling rate. Calibration estimates these from experimental data on phenology, biomass and yield, usually by adjusting parameters to minimise the discrepancy between simulated and observed values, by trial and error, by automated optimisation, or by Bayesian methods that give a posterior distribution of the parameters (approaches such as GLUE, generalised likelihood uncertainty estimation, and Markov chain Monte Carlo). Sensitivity analysis identifies which parameters matter most for the output of interest, so that effort goes to those. A key principle is to calibrate on one data set and evaluate on another, independent one (different sites, seasons or treatments); a model judged on its calibration data will look better than it is.
Simulated and observed values are compared with several statistics, each answering a different question. The root mean square error, RMSE = square root of the mean of (simulated - observed) squared, gives the typical error in data units; the normalised RMSE divides it by the observed mean and is expressed as a percentage, with values below about 10 per cent often called excellent and 10 to 20 per cent good, although the bands are conventions rather than standards. The mean bias shows systematic over- or under-prediction. Willmott's index of agreement d lies between 0 and 1, with 1 meaning perfect agreement. The Nash-Sutcliffe efficiency, 1 - sum (O - S) squared / sum (O - O-bar) squared, is 1 for a perfect model, 0 when the model is no better than the observed mean, and negative when it is worse. The regression of observed on simulated values (slope 1, intercept 0 ideal) and the coefficient of determination indicate association but not agreement, because a model can be perfectly correlated with observations and still be badly biased.
Simulation models are built for a field with known inputs, but forecasts are needed for districts and countries where soils, sowing dates, cultivars and management vary and are not known in detail. Upscaling strategies run the model on a grid of representative combinations and aggregate, or run it with regional average inputs, and then correct the output statistically against reported yields: a regression of district yield on simulated yield and a trend, for example. Data assimilation updates model states such as leaf area index during the season with remote-sensing observations, using methods such as the ensemble Kalman filter (introduced in Time Series Analysis) or by re-estimating parameters, so that the simulation stays close to what the satellite sees. The uncertainty from inputs, parameters and model structure is large, and ensembles of several models are increasingly used to express it.
Machine-learning (ML) methods learn the relation between yield and many predictors without specifying its form in advance. Those commonly used for yield forecasting include penalised regressions (ridge, lasso, elastic net), regression trees and their ensembles (random forests, gradient-boosted trees), support-vector regression, artificial neural networks, and deep-learning models such as convolutional networks on satellite images and recurrent networks (for example long short-term memory, LSTM) on weather and vegetation-index time series. Their strengths are flexibility with many correlated predictors, the ability to capture non-linear and interaction effects (such as heat stress mattering only when water is short), and good use of large spatial data sets. Penalised regression and tree ensembles are often strong baselines, and complex deep models do not automatically beat them on the modest data sets typical of agricultural statistics.
The main risk of ML is overfitting and over-optimistic evaluation. Yield data are correlated in time (a district's yields share a trend) and in space (neighbouring districts share weather). Random k-fold cross-validation mixes years and neighbours across training and test folds, so the model is tested on data it has effectively seen; reported accuracy is then inflated. Honest validation leaves out whole years (leave-one-year-out or forward-in-time validation, training only on years before the test year) and, where the aim is to forecast new areas, whole spatial blocks. Data leakage also occurs when a predictor contains information not available at forecast time, such as end-of-season NDVI used in a mid-season forecast, or a trend fitted on all years including the test year. Hyperparameters must be tuned inside the training folds only.
Hybrid models combine statistical, process-based and ML components. Examples are ARIMA plus a neural network, where ARIMA models the linear time-series structure and the network models the residual non-linear part (a design popularised by Zhang in 2003); ML models that take outputs of crop simulation models (simulated biomass, water stress) as predictors alongside weather and remote-sensing data; and simulation models whose parameters are estimated by ML. Hybrids often improve accuracy because each component captures something the others miss. Because policy users need to trust and explain forecasts, interpretation matters: variable-importance measures, partial-dependence plots and Shapley-value explanations show which predictors drive a forecast, and a forecast that rests on a physiologically implausible predictor should be treated with suspicion even if it scores well.
A crop forecast is judged against the final estimate made after harvest, usually from crop-cutting surveys, or against the official final figure. The basic measure is the percentage deviation, (forecast - final)/final x 100, computed for each year and region. Across years, the mean absolute percentage error, the root mean square error and the mean bias summarise accuracy; a forecast that is consistently too high is a different problem from one that is unbiased but noisy. Accuracy should be reported separately for each forecast date, since it improves as harvest nears, and for each spatial level, since errors partly cancel when districts are aggregated to provinces or states. Comparisons should be made against simple benchmarks, such as the trend value or last year's yield, because a model that cannot beat the trend adds nothing. These measures and formal tests for comparing forecasts are treated in Time Series Analysis.
Many governments publish a sequence of estimates for each season: early forecasts based on area and weather, then advance estimates updated with survey and remote-sensing information, then final estimates. For example, the United States Department of Agriculture issues monthly crop production estimates through the season, the European Commission's MARS bulletins publish monthly yield forecasts for EU member states, and India's agriculture ministry releases a series of advance estimates followed by final estimates; the number and timing of releases change over time, so current practice should be checked. Revisions between successive estimates are a natural part of the process, but large or systematic revisions are a warning: if early estimates are always revised downwards, users learn to discount them. Publishing the record of past revisions and errors, alongside the forecast, helps users weigh it properly.
Forecasts inform procurement targets, the release of stocks, decisions on export restrictions and import duties, planning of storage and transport, and early warning of food insecurity. Crop insurance also relies on yield estimates: area-based schemes in which payouts depend on the yield of an insurance unit need yields estimated for many units quickly and credibly, and technology-based estimates using remote sensing and models are increasingly used alongside, or to support, crop-cutting experiments, with rules that differ by country and scheme. In each use, the cost of an error differs: overestimating a crop may lead to exports that later force expensive imports, while underestimating it may trigger unnecessary imports that depress farm prices. A forecaster should therefore report uncertainty, explain what the forecast rests on, and keep a public record of accuracy.
Forecasts that affect prices and payments attract pressure. Good practice separates the statistical agency producing forecasts from the departments that use them, documents methods, releases on a pre-announced calendar, and gives all users the information at the same time. International monitoring by FAO, the AMIS partners and others provides an independent check on national figures for major traded crops. Statistically, the safeguards are those already described: models validated on independent years, multiple sources combined, benchmarks reported, and errors published. A forecasting system earns trust by being right often enough and honest about when it was wrong.
A student must be able to explain why forecasts are needed before harvest and what information each model type uses, write the weighted weather-index formula and say what the weights are, compute NDVI from reflectances, growing degree-days from temperatures and a percentage forecast deviation, recognise the logistic, Gompertz, monomolecular and Richards curves from their equations and inflection points, and describe how a non-linear model is fitted by Gauss-Newton or Marquardt iterations. In a thesis the student fits and validates a district forecast model or a growth curve and defends the choice of model and validation design.
Five years into work at a state agriculture department, a crop forecasting centre or an agribusiness, the statistician builds and updates district models each season: assembling weekly weather, satellite indices and survey data, refitting weather-index and composite models, checking them on held-out years, and producing forecasts with error ranges on a fixed calendar. The practitioner deals with missing station data, delayed survey returns and cloud-covered imagery, and explains to managers why the early forecast changed.
A senior official or research manager with 25 years of experience decides how a national forecasting system combines surveys, models and satellites, sets the release calendar, and defends forecasts that affect procurement, trade and insurance payments. The questions at this level concern governance and long-run performance: whether early estimates are biased, whether a change of satellite or survey design broke a series, how forecasts and their errors are communicated to ministers and markets, and how pressure on the numbers is resisted.
A professor or research leader works on the methods themselves: Bayesian and hierarchical models that borrow strength across districts and years, data assimilation of satellite observations into crop simulation models, ensembles of process and statistical models, machine-learning models with honest spatial and temporal validation, and probabilistic forecasts that express uncertainty. The research leader supervises theses, reviews forecasting papers whose accuracy claims often rest on leaky validation, and advises agencies on adopting new methods responsibly.
This manual describes forecasting programmes (FASAL, MARS, GIEWS, NASS) only in outline; their current procedures, release schedules and accuracy records change and must be read from the agencies' own documents. Crop-specific base temperatures, critical stages and model coefficients are not given because they vary by crop, cultivar and region. Small-area estimation for district forecasts, Bayesian hierarchical forecasting, and the detailed algorithms of crop simulation models and deep-learning architectures are named but not developed. Accuracy bands for normalised RMSE are conventions from the modelling literature, not standards.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions. Every numerical answer was computed with a script; the values needed are stated in the question.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.