Agricultural Statistics & Data Science · Statistical computing, data science and practice · also Machine Learning, Statistical Learning, Data Science
Prediction from data with honest validation: the bias-variance trade-off, regularised regression, trees and ensembles, support vector machines, neural networks and deep learning for images, unsupervised learning, and how to use them on yield, disease, price and image data without fooling yourself. The Agribusiness and Agricultural Engineering tracks introduce ML; this is the statistician's full course.
In the exams: GATE DA 'Machine Learning' and 'AI' (heavy); AICE-JRF Computer Application (present); SAS Statistics for Machine Learning credential (present)
Machine learning is the part of statistics concerned with prediction: learning from data a rule that predicts an outcome for new cases. A model that predicts district wheat yield from weather and satellite data, one that recognises leaf rust in a phone photograph, one that flags milk samples likely to be adulterated, and one that groups farms by management style are all machine-learning models. The methods range from regularised regression, which any statistician recognises, through decision trees and their ensembles, support vector machines and neural networks, to unsupervised methods that find structure without an outcome.
The central idea of the subject is honest validation. A flexible model can fit its training data almost perfectly and still predict new data badly. The bias-variance trade-off explains why; cross-validation measures it; and in agriculture, where observations are correlated in space and time, cross-validation must be blocked by field, district or year or it will mislead. This manual therefore treats validation early and returns to it in every chapter. It also covers interpretation (variable importance, partial dependence and SHAP values), because a forecast that cannot be explained will not be trusted, and the foundations of artificial intelligence (search, logic and reasoning under uncertainty) that some examinations include.
In examinations, machine learning and artificial intelligence are heavy sections of data-science papers such as GATE Data Science and Artificial Intelligence, and appear in statistical computing and professional credentials. In agricultural research they are now routine in yield forecasting, remote sensing, plant phenotyping, disease detection and precision agriculture. Classical multivariate methods that these build on (principal components, discriminant and cluster analysis) are treated in this track's Multivariate Analysis; crop-yield applications in Crop Forecasting and Growth Models; spatial validation and image classification in Spatial Statistics and Remote Sensing; and field applications in Precision Agriculture and System Management in the Agricultural Engineering track.
In supervised learning we observe pairs (x, y): a vector of predictors (features) x and an outcome y. When y is numerical (yield, price, milk fat percentage) the task is regression; when y is a category (diseased or healthy, crop type) it is classification. We seek a function f such that f(x) predicts y well for new cases drawn from the same population. The quality of a model is judged by its expected loss on new data: squared error for regression, misclassification rate or log loss for classification. The training error, the loss on the data used to fit the model, always understates this, because the model has adapted to the particular sample, including its noise. Two goals should be kept apart: prediction, where only accuracy on new cases matters, and inference, where we want to understand how each predictor relates to y. Many machine-learning methods are excellent for prediction and poor for inference, and a model that predicts well need not reflect causation.
For squared-error loss, the expected prediction error at a point x0 decomposes into three parts: E[(y0 - f-hat(x0)) squared] = sigma squared + [Bias(f-hat(x0))] squared + Var(f-hat(x0)). The irreducible error sigma squared is the noise in y that no model can remove. Bias is the error from approximating a complex reality by a simpler model, for example fitting a straight line to a curved response of yield to nitrogen. Variance is the amount by which the fitted model would change if it were fitted to a different training sample. Flexible models (deep trees, small-k nearest neighbours, high-degree polynomials, large neural networks) have low bias and high variance; rigid models (linear regression with few predictors) have high bias and low variance. As flexibility increases, training error falls steadily, while test error typically falls and then rises, a U-shape. The best model sits at the bottom of the test-error curve, balancing the two.
Overfitting means fitting the noise: the model captures accidental patterns of the training sample that do not recur. It is most likely when there are many predictors relative to observations, a common situation in agriculture: thirty years of district yields and hundreds of weekly weather and satellite variables, or a few hundred plants and thousands of genetic markers. The tools to control it are restricting flexibility (fewer predictors, shallower trees), regularisation (penalties that shrink parameters, Chapter 3), averaging many models (bagging and random forests, Chapter 6), early stopping in iterative fitting, and above all measuring performance on data not used for fitting. The amount of data matters too: more data reduce variance and let more flexible models be used safely. A simple model that is well validated is usually more useful than a complex model that is not.
Good practice separates three roles. The training set is used to fit model parameters. The validation set is used to choose among models and tune hyperparameters (settings fixed before fitting, such as the penalty in ridge regression or the depth of a tree). The test set is used once, at the end, to estimate the performance of the final chosen model. If the test set influences any choice, its estimate becomes optimistic. With small data sets, cross-validation (Chapter 2) replaces the single validation set. Every preprocessing step that learns from data, such as standardising predictors, imputing missing values or selecting features, must be fitted on the training part only and then applied to validation and test data; doing it on the whole data set leaks information and inflates accuracy.
Cross-validation estimates test error by repeatedly holding out part of the data. In k-fold cross-validation the data are divided at random into k folds of about equal size; for each fold, the model is fitted on the other k - 1 folds and its error is computed on the held-out fold; the k errors are averaged. Common choices are k = 5 or 10. Each observation is used for testing exactly once. Cross-validation is used both to estimate the performance of a given model and to choose hyperparameters: for each candidate value of the penalty or tree depth, the cross-validated error is computed and the value with the smallest error is chosen. The one-standard-error rule picks the simplest model whose cross-validated error is within one standard error of the minimum, favouring parsimony when differences are within noise.
Leave-one-out cross-validation (LOOCV) is k-fold with k = n: each observation is held out in turn. It uses almost all data for each fit, so its estimate of test error has low bias, but the n fitted models are nearly identical, so their errors are highly correlated and the estimate can have high variance; it is also costly for large n. For linear regression fitted by least squares there is a shortcut: the LOOCV error equals the average of [e(i)/(1 - h(i))] squared, where e(i) is the ordinary residual and h(i) the leverage of observation i, so no refitting is needed. In practice 5- or 10-fold cross-validation often gives a better bias-variance balance for the error estimate than LOOCV.
When cross-validation is used to tune hyperparameters, the minimum cross-validated error is an optimistic estimate of the tuned model's performance, because the tuning chose the value that happened to do best on those folds. Nested cross-validation separates the two jobs: an outer loop holds out folds for evaluation, and within each outer training set an inner cross-validation loop tunes the hyperparameters. The outer folds never influence tuning, so their average error is a fair estimate of how the whole procedure, tuning included, will perform. Feature selection is part of the procedure too and must be repeated inside each training fold; selecting predictors on all data and then cross-validating is a classic error that produces impressive but false accuracy.
Random folds assume observations are independent. Agricultural data rarely are. Yields of neighbouring districts share weather; plots in the same field share soil; pixels in the same field share everything; consecutive years share trends. With random folds, the test fold contains near-copies of training observations, and the model is rewarded for memorising local conditions rather than learning relationships that transfer. The remedy is blocked cross-validation: hold out whole spatial blocks (fields, villages, districts, or square blocks larger than the range of spatial autocorrelation, see Spatial Statistics and Remote Sensing) when the aim is to predict new places, and hold out whole years, or train only on earlier years and test on later ones (forward-chaining), when the aim is to forecast future seasons. Grouped cross-validation keeps all records of one farm, animal or plant in the same fold. The validation design should match the use: a model for new districts needs spatial blocks; a model for next year needs temporal blocks.
Ordinary least squares is unbiased, but with many correlated predictors its estimates have large variance, and when there are more predictors than observations it has no unique solution. Regularisation accepts a little bias in exchange for a large reduction in variance by adding a penalty on the size of the coefficients to the residual sum of squares. The penalty strength lambda is a hyperparameter chosen by cross-validation: lambda = 0 gives least squares, and very large lambda shrinks coefficients towards zero. Because the penalty treats all coefficients alike, predictors must be standardised (centred and scaled to unit variance) before fitting, or the penalty will depend on their units. The intercept is not penalised.
Ridge regression minimises RSS + lambda times the sum of beta(j) squared, an L2 penalty. It has the closed-form solution beta-hat = (X'X + lambda I) inverse X'y, which exists even when X'X is singular. Ridge shrinks all coefficients towards zero but never sets any exactly to zero, so it keeps every predictor. It is particularly effective with many correlated predictors, sharing weight among them rather than picking one arbitrarily. In the special case of orthonormal predictors, ridge simply divides each least-squares coefficient by (1 + lambda). Ridge regression is closely related to the best linear unbiased prediction of random effects: genomic BLUP (GBLUP) and ridge regression on markers are essentially the same model, which links this chapter to statistical genetics and genomic selection.
The lasso (least absolute shrinkage and selection operator, Tibshirani 1996) minimises RSS + lambda times the sum of the absolute values |beta(j)|, an L1 penalty. Because of the corners of the L1 constraint region, the lasso sets some coefficients exactly to zero, performing variable selection and giving sparse, interpretable models: from 200 weather indices it may keep 12. In the orthonormal case it soft-thresholds each least-squares coefficient, shrinking it towards zero by a fixed amount and setting it to zero if it is smaller than that amount. The lasso has weaknesses: among a group of highly correlated predictors it tends to pick one more or less arbitrarily, and when predictors outnumber observations it selects at most n of them. The elastic net combines both penalties, lambda[alpha times the L1 penalty + (1 - alpha)/2 times the L2 penalty] in a common parameterisation, keeping the lasso's sparsity while selecting correlated predictors together. Alpha is chosen by cross-validation along with lambda.
Penalised regression is often the first model to try on high-dimensional agricultural data: yield on many weather and remote-sensing variables, near-infrared spectra predicting soil or forage quality (hundreds of wavelengths, strongly correlated), and genomic prediction from thousands of markers. It is fast, stable and easy to interpret, and it is a strong baseline that complex methods must beat. Two cautions apply. The coefficients selected by the lasso should not be read as the causal drivers: with correlated predictors, a different sample might select different ones. And standard errors and p-values from a naive refit of the selected model are invalid, because selection used the same data; post-selection inference requires special methods. Partial least squares and principal component regression (see Multivariate Analysis) are alternatives that also handle many correlated predictors.
The k-nearest neighbours (kNN) method predicts for a new case by finding the k training cases closest to it in predictor space, usually by Euclidean distance, and taking the majority class among them (classification) or the average of their outcomes (regression). It makes no assumption about the form of the relationship and needs no fitting step; all the work happens at prediction time. The choice of k controls the bias-variance trade-off: k = 1 gives a very flexible, high-variance rule whose training error is zero; large k gives a smooth, high-bias rule; k is chosen by cross-validation, and odd values avoid ties in two-class problems. Because distances depend on units, predictors must be standardised; otherwise a variable measured in kilograms dominates one measured in tonnes. Weighting neighbours by inverse distance is a common refinement.
kNN works well with a few predictors and many observations but degrades quickly as the number of predictors grows. In high dimensions, all points are far from one another, and the nearest neighbour is hardly nearer than the farthest, so neighbourhoods stop being local. To capture a fixed fraction of the data, a neighbourhood must cover a large part of the range of every predictor. This curse of dimensionality affects all local methods, including kernel smoothers, and is one reason why dimension reduction (Chapter 10) or methods that select relevant predictors are used before local methods. Irrelevant predictors are particularly harmful to kNN, because they add noise to every distance.
Naive Bayes classifiers apply Bayes' theorem, P(class | x) proportional to P(class) times P(x | class), with the simplifying (naive) assumption that the predictors are conditionally independent given the class, so that P(x | class) is the product of the individual P(x(j) | class). For continuous predictors each factor is often a normal density with a class-specific mean and variance (Gaussian naive Bayes); for counts, a multinomial or Poisson model; for binary features, Bernoulli. The independence assumption is usually false, yet naive Bayes often classifies well, because classification needs only the correct ranking of posterior probabilities, not accurate probabilities themselves. It is fast, works with small training sets and many features, and handles missing predictors naturally. Zero counts for a feature in a class would make the product zero; Laplace (add-one) smoothing prevents this. Its probability estimates tend to be overconfident and should not be used as calibrated risks without adjustment.
Discriminant analysis classifies by modelling the distribution of predictors within each class and applying Bayes' rule. Linear discriminant analysis (LDA) assumes that in each class x is multivariate normal with a class-specific mean vector and a covariance matrix common to all classes; the resulting decision boundaries are linear in x. Quadratic discriminant analysis (QDA) allows each class its own covariance matrix, giving quadratic boundaries. QDA is more flexible but estimates many more parameters, p(p + 1)/2 covariance terms per class instead of once in total, so it needs more data; LDA can outperform QDA when samples are small even if covariances differ. With equal priors and two classes, LDA reduces to Fisher's linear discriminant function, which maximises the ratio of between-class to within-class variance (see Multivariate Analysis). Regularised discriminant analysis shrinks QDA towards LDA. These are the same rules as the maximum-likelihood classifier for satellite images.
Logistic regression models the probability of class 1 directly: log[p/(1 - p)] = beta0 + beta1 x1 + ... + betap xp, so p = 1/(1 + exp(-(beta0 + beta' x))). The left side is the log-odds; exp(beta(j)) is the odds ratio for a one-unit increase in x(j), holding the others fixed. Parameters are estimated by maximum likelihood (iteratively reweighted least squares), not by least squares. Logistic regression makes no assumption about the distribution of the predictors and gives directly interpretable effects; its decision boundary is linear, like LDA's. When the classes are well separated, logistic estimates can become unstable or infinite (complete separation), while LDA remains stable; when predictors are far from normal, logistic regression is usually preferable. Penalised (ridge or lasso) logistic regression handles many predictors. Multinomial logistic regression extends it to more than two classes. The broader family is treated in Categorical Data and Generalised Linear Models.
A classifier's predictions are summarised in a confusion matrix of predicted against actual classes. For a two-class problem with a positive class (say, diseased), sensitivity (recall, true-positive rate) is the proportion of actual positives correctly predicted, specificity is the proportion of actual negatives correctly predicted, and precision (positive predictive value) is the proportion of predicted positives that are truly positive. The F1 score is the harmonic mean of precision and recall. Accuracy alone misleads when classes are imbalanced: if 2 per cent of animals are diseased, a rule that calls every animal healthy is 98 per cent accurate and useless. The receiver operating characteristic (ROC) curve plots sensitivity against 1 - specificity as the probability threshold varies; the area under it (AUC) is the probability that a randomly chosen positive is scored higher than a randomly chosen negative, 0.5 for random guessing and 1 for perfect ranking. The decision threshold should reflect the costs of each error, not default to 0.5.
A decision tree splits the predictor space into rectangles by a sequence of binary questions (Is rainfall in July below 210 mm? Is soil pH above 7.5?) and predicts by the mean outcome (regression) or the majority class (classification) in each final region, or leaf. The CART algorithm of Breiman and colleagues grows the tree greedily: at each node it chooses the predictor and cut-point that most reduce the residual sum of squares (regression) or an impurity measure (classification). Common impurity measures are the Gini index, 1 minus the sum of p(k) squared over classes, and entropy, minus the sum of p(k) log p(k); both are zero for a pure node and largest when classes are equally mixed. A large tree overfits, so it is pruned back by cost-complexity pruning, which penalises the number of leaves with a parameter chosen by cross-validation. Trees are easy to explain and handle mixed predictor types, missing values and interactions naturally, but a single tree has high variance: a small change in the data can change the whole tree.
Bagging (bootstrap aggregation) reduces the variance of trees by averaging many of them: draw B bootstrap samples (samples of size n with replacement), grow a deep tree on each, and average their predictions (or take a majority vote). Each bootstrap sample omits on average about 36.8 per cent of the observations (the limit of (1 - 1/n) to the power n is 1/e), and these out-of-bag observations give a free estimate of test error. If a few predictors are very strong, the bagged trees are highly correlated and averaging helps less. Random forests (Breiman, 2001) decorrelate the trees by considering only a random subset of m predictors at each split; common defaults are m about the square root of p for classification and p/3 for regression. Random forests work well with little tuning, resist overfitting as the number of trees grows, and are a strong default for tabular agricultural data. They extrapolate poorly: a forest cannot predict a yield higher than any in its training data.
Boosting builds trees sequentially, each one fitted to correct the errors of the ensemble so far. In gradient boosting (Friedman), each new small tree is fitted to the negative gradient of the loss, which for squared error is simply the current residuals, and is added to the model multiplied by a small learning rate (shrinkage), for example 0.01 to 0.1. The main hyperparameters are the number of trees, the learning rate and the depth of each tree (often 2 to 8 splits, controlling the order of interactions), with subsampling of rows and columns adding randomness. Unlike random forests, boosting can overfit if too many trees are added, so the number of trees is chosen by cross-validation or early stopping. A smaller learning rate needs more trees but usually generalises better. Tuned gradient boosting is often the most accurate method on tabular data, but it is more sensitive to tuning than random forests, and both require the honest, blocked validation of Chapter 2.
For two classes that can be separated by a hyperplane, the maximal margin classifier chooses the separating hyperplane farthest from the nearest training points; the distance is the margin, and the points on the margin are the support vectors. Only the support vectors determine the hyperplane; moving other points (without crossing the margin) does not change it. Real data are rarely separable, so the support vector classifier allows some points to violate the margin or be misclassified, controlled by a cost or budget parameter C. In the common parameterisation in which C multiplies the penalty for violations, a large C tolerates few violations, giving a narrow margin, low bias and high variance; a small C allows many violations and a wide margin, with higher bias and lower variance. C is chosen by cross-validation. The fitting problem is a convex quadratic optimisation with a unique solution, and it can be written as minimising the hinge loss plus a ridge-type penalty, which connects SVMs to regularised logistic regression.
Linear boundaries are often inadequate. Support vector machines enlarge the feature space implicitly through kernels. The solution depends on the data only through inner products between observations, and replacing the inner product by a kernel function K(x, x') computes the boundary in a high-dimensional (even infinite-dimensional) feature space without constructing it: the kernel trick. The polynomial kernel (1 + x'x') to the power d gives polynomial boundaries; the radial basis function (Gaussian) kernel exp(-gamma times the squared distance between x and x') gives very flexible, local boundaries, with gamma controlling how local: large gamma, wiggly boundaries and high variance. The kernel parameters and C are tuned jointly by cross-validation, usually on a grid. Predictors must be standardised. SVMs handle more than two classes by combining binary classifiers, one-versus-one or one-versus-rest.
Support vector regression (SVR) fits a function that keeps most training points within a tube of width epsilon around it, ignoring errors smaller than epsilon (the epsilon-insensitive loss) and penalising larger ones linearly; only points on or outside the tube are support vectors. With kernels it fits non-linear relationships. SVMs were widely used in agricultural remote sensing (crop classification with many spectral bands), in spectroscopy and in disease detection from image features, because they perform well with many predictors and modest sample sizes. Their drawbacks are that they do not give probabilities directly (these must be calibrated separately), they scale poorly to very large data sets, and kernel models are hard to interpret. For tabular data they have often been overtaken by tree ensembles, and for images by convolutional networks.
A feed-forward neural network (multilayer perceptron) transforms the input through layers of units. Each unit computes a weighted sum of its inputs plus a bias and applies a non-linear activation function: the sigmoid, the hyperbolic tangent, or most commonly in modern networks the rectified linear unit, ReLU(z) = max(0, z). Without non-linear activations, any number of layers would collapse into a single linear model. The output layer uses an identity activation for regression and a softmax for classification, giving class probabilities that sum to 1. A network with one sufficiently wide hidden layer can approximate any continuous function (the universal approximation theorem), but deeper networks often represent complex functions with far fewer units. The number of parameters grows quickly: a fully connected layer from 100 inputs to 50 units has 100 x 50 weights plus 50 biases, 5,050 parameters.
Networks are trained by minimising a loss (squared error, or cross-entropy for classification) with gradient descent. Backpropagation computes the gradient of the loss with respect to every weight efficiently by applying the chain rule backwards from the output. Stochastic gradient descent updates weights using small random mini-batches of data; one pass through the training data is an epoch. The learning rate sets the step size: too large and training diverges, too small and it crawls. Variants such as momentum and adaptive methods adjust steps automatically. Because networks have many parameters, they overfit easily, and regularisation is essential: weight decay (an L2 penalty), dropout (randomly switching off units during training), data augmentation, and early stopping, which halts training when error on a validation set starts to rise. Results depend on random initialisation, so performance should be reported over several runs.
Images have spatial structure that fully connected networks ignore. Convolutional neural networks (CNNs) use convolutional layers in which small filters (for example 3 x 3 pixels) slide across the image, computing the same weighted sum at every position. Parameter sharing makes the layer far smaller than a dense one and lets it detect a feature (an edge, a lesion texture) wherever it appears. A convolutional layer with k x k filters, c(in) input channels and c(out) filters has (k x k x c(in) + 1) x c(out) parameters. The output width of a convolution is (W - K + 2P)/S + 1 for input width W, filter size K, padding P and stride S. Pooling layers (for example 2 x 2 max pooling) reduce spatial size. Stacked layers learn increasingly abstract features, from edges to textures to whole leaves. Transfer learning, starting from a network pre-trained on a large general image collection and fine-tuning it on a smaller agricultural data set, is the standard way to build image models when labelled agricultural images are limited.
CNNs are used to identify plant diseases and pests from leaf photographs, count fruits, heads or plants, detect weeds for precision spraying, score phenotypes in breeding trials, assess carcass or animal body condition, and classify crops in satellite and drone images. Results reported on curated data sets can be very high and still fail in the field. Images collected in a laboratory against a plain background do not represent field photographs with variable light, soil and overlapping leaves; a model can learn the background, the camera or the photographer rather than the disease. Random splits of images that include several photographs of the same plant or field put near-duplicates in training and test sets. Honest evaluation tests on images from different fields, seasons, devices and regions, and reports performance for each class, since rare diseases matter most. Labelling errors by annotators also limit attainable accuracy.
Unsupervised learning looks for structure without an outcome variable. Clustering groups observations so that those in the same group are similar. k-means partitions n observations into k clusters to minimise the total within-cluster sum of squared Euclidean distances to the cluster means. Lloyd's algorithm alternates two steps: assign each observation to the nearest centroid, then recompute each centroid as the mean of its members; the objective never increases, and the algorithm converges, but only to a local minimum that depends on the starting centroids. It is therefore run from many random starts (or a careful initialisation such as k-means++), keeping the best solution. k-means favours compact, roughly spherical clusters of similar size, is sensitive to outliers and to the scale of variables (standardise first), and requires k to be chosen.
k-medoids, often fitted by the PAM (partitioning around medoids) algorithm, is like k-means but represents each cluster by a medoid, an actual observation that minimises the total dissimilarity to the other members, rather than by a mean. It can use any dissimilarity, not only Euclidean distance, which matters for farm-survey data that mix numerical, ordinal and categorical variables: Gower's dissimilarity combines them by scaling each variable's contribution to between 0 and 1. Because it minimises sums of dissimilarities rather than squared distances, k-medoids is more robust to outliers. The medoids are real cases, which makes clusters easy to describe: this farm is typical of cluster 3. Its cost is computation, which grows faster with n than k-means.
Agglomerative hierarchical clustering starts with each observation as its own cluster and repeatedly merges the two closest clusters until one remains, recording the sequence in a dendrogram, a tree whose merge heights show the dissimilarity at which clusters joined. Cutting the dendrogram at a chosen height gives any number of clusters, so k need not be fixed in advance. The result depends on the linkage, the definition of distance between clusters: single linkage (nearest pair) tends to produce long chains; complete linkage (farthest pair) gives compact clusters; average linkage uses the mean of all pairwise distances; Ward's method merges the pair that least increases the total within-cluster sum of squares and behaves like k-means. Hierarchical clustering of genotypes by Mahalanobis D-squared or Euclidean distance on traits is standard in plant breeding for grouping divergent parents (see Multivariate Analysis).
There is no single correct number of clusters. Common aids are the elbow plot (within-cluster sum of squares against k, looking for a bend), the average silhouette width (for each observation, s = (b - a)/max(a, b), where a is its mean distance to its own cluster and b to the nearest other cluster; values near 1 indicate good separation, near 0 ambiguous assignment, and negative values likely misassignment) and the gap statistic, which compares within-cluster dispersion with that expected under a reference distribution with no clusters. Clustering algorithms always produce clusters, even from random data, so stability should be checked by re-running on bootstrap samples or subsets, and clusters should be judged by whether they are useful and interpretable, for example farm typologies that predict adoption of a technology.
Principal component analysis (PCA) replaces p correlated variables by a smaller number of uncorrelated linear combinations, the principal components, ordered by the variance they explain. The components are the eigenvectors of the covariance matrix (or of the correlation matrix when variables are standardised), and the variance of each component is the corresponding eigenvalue. The proportion of variance explained by component j is its eigenvalue divided by the sum of all eigenvalues, which equals p when the correlation matrix is used. PCA on unstandardised variables is dominated by those with large variances, so variables in different units should be standardised. The number of components to keep is chosen from a scree plot, a cumulative proportion of variance (for example 80 to 90 per cent), the rule of keeping components with eigenvalue above 1 for a correlation matrix, or, in a predictive setting, by cross-validation. The full theory is in Multivariate Analysis; here PCA serves as a preprocessing step and a visual tool.
Principal component regression regresses the outcome on the first few components, avoiding multicollinearity, but the components are chosen to explain the predictors, not the outcome, so a component with small variance may be the one that matters. Partial least squares instead builds components that have high covariance with the outcome, which is why it is standard for near-infrared spectra predicting soil carbon, forage protein or grain moisture. In any predictive use, PCA must be fitted on the training data only and the same loadings applied to test data; fitting PCA on all data before cross-validation leaks information. PCA is linear: it cannot unfold a curved structure, and it is sensitive to outliers, for which robust versions exist.
Embeddings map high-dimensional data to a low-dimensional space while trying to keep similar observations close. t-distributed stochastic neighbour embedding (t-SNE) and UMAP are widely used to visualise data such as gene expression, hyperspectral pixels or image features in two dimensions. They preserve local neighbourhoods well but distort global structure: distances between clusters, cluster sizes and densities in a t-SNE plot are not meaningful, and the picture changes with tuning parameters (perplexity for t-SNE, number of neighbours for UMAP) and random seeds. They are tools for exploration, not for measuring distances or proving that groups exist. Autoencoders, neural networks trained to reconstruct their input through a narrow middle layer, learn non-linear embeddings that can serve as features for other models. Learned embeddings of categories (for example of varieties, districts or farms) let neural networks represent many-level categorical variables compactly.
A yield forecast that will set procurement targets, or a disease classifier that will guide treatment, must be trusted by people who did not build it. Interpretation serves three purposes: to check that the model relies on sensible predictors and not on artefacts (a field identifier, the date an image was taken, a leaked post-harvest variable), to communicate how predictions arise, and to generate hypotheses. It does not establish causation: interpretation describes the model, and the model describes associations in the data. With correlated predictors, credit can be shared or shifted between them in ways that do not reflect any physical mechanism.
Tree ensembles provide impurity-based importance: the total reduction in RSS or Gini impurity from splits on each predictor, averaged over trees. It is cheap but biased towards predictors with many possible split points (continuous variables, categorical variables with many levels), even if they are noise. Permutation importance measures how much the model's error increases when one predictor's values are randomly shuffled, breaking its relation with the outcome; computed on held-out or out-of-bag data, it is model-agnostic and less biased, but when predictors are correlated, shuffling one creates unrealistic combinations and the importance of a group may be split or understated. Conditional and grouped permutation schemes address this. Importance ranks predictors; it does not show the direction or shape of their effects.
A partial dependence plot shows how the model's average prediction changes as one predictor varies over its range, averaging over the observed values of the others. It reveals the shape of an effect, for example yield rising with rainfall and then falling under waterlogging. Individual conditional expectation (ICE) curves show the same for each observation separately, revealing interactions when curves are not parallel. Both assume that the predictor can be varied independently of the others; with strongly correlated predictors they evaluate the model in regions with no data. Accumulated local effects plots reduce this problem by working with local changes within the observed data.
SHAP (SHapley Additive exPlanations, Lundberg and Lee, 2017) attributes each individual prediction to the predictors using Shapley values from cooperative game theory. Each predictor's SHAP value is its average marginal contribution to the prediction over all possible orders in which predictors could be added. The values are additive: for each observation, the baseline (average prediction) plus the sum of the SHAP values equals the model's prediction. Summaries across observations (mean absolute SHAP values, or beeswarm plots showing the value of each predictor against its SHAP value) give global importance and direction. Exact computation is exponential in the number of predictors, but efficient algorithms exist for tree ensembles, and approximations for other models. SHAP values inherit the cautions above: they explain the model, not the world, and with correlated predictors the attribution depends on how missing predictors are handled. LIME, another local method, fits a simple model around one prediction; it is easy to use but its explanations can be unstable.
Many AI problems are posed as search through a space of states, from an initial state to a goal, by applying actions with costs: route planning for a harvesting machine, scheduling irrigation, solving a puzzle. Uninformed search uses no knowledge of the goal's location. Breadth-first search expands the shallowest nodes first; it finds the shallowest goal and is optimal when all step costs are equal, but its memory grows exponentially with depth. Depth-first search expands the deepest node first; it uses little memory but may follow an infinite path and is neither complete in infinite spaces nor optimal. Uniform-cost search expands the node with the lowest path cost g(n) and is optimal for non-negative costs. Informed search uses a heuristic h(n) estimating the cost from n to the goal. Greedy best-first search expands the node with the smallest h(n) and is fast but not optimal. A* search expands the node with the smallest f(n) = g(n) + h(n); with an admissible heuristic (one that never overestimates the true cost) A* tree search is optimal, and with a consistent heuristic so is graph search. Adversarial search for two-player games uses minimax, with alpha-beta pruning to skip branches that cannot affect the decision.
Logic represents knowledge as sentences and derives new sentences by inference. In propositional logic, sentences combine propositions with connectives (not, and, or, implies, if and only if); a sentence is valid if true in every model and satisfiable if true in some. Entailment (a knowledge base KB entails alpha) means alpha is true in every model in which KB is true. Sound inference rules derive only entailed sentences: modus ponens (from alpha implies beta, and alpha, infer beta), and resolution, which is complete for refutation when sentences are in conjunctive normal form: to prove alpha, add not-alpha to KB and derive the empty clause. First-order (predicate) logic adds objects, relations, functions and the quantifiers 'for all' and 'there exists', so that a rule such as 'every field with standing water and a susceptible variety is at risk of blast' can be stated once for all fields. Inference in first-order logic uses unification and generalised modus ponens or resolution; it is semi-decidable. Rule-based expert systems for crop diagnosis are applications of these ideas.
Real decisions are uncertain, and probability is the language of uncertain reasoning. Bayes' rule, P(cause | evidence) = P(evidence | cause) P(cause)/P(evidence), updates beliefs with evidence: given a disease prevalence of 1 per cent, a test with 90 per cent sensitivity and 95 per cent specificity, a positive result raises the probability of disease only to about 15 per cent. A Bayesian network is a directed acyclic graph whose nodes are random variables and whose edges encode direct dependence; each node has a conditional probability table given its parents, and the joint distribution is the product of these tables. The graph encodes conditional independence: each node is independent of its non-descendants given its parents, which makes large joint distributions compact. Exact inference computes posterior probabilities by enumeration or, more efficiently, by variable elimination, which sums out variables one at a time in a chosen order; exact inference is intractable in general. Approximate inference uses sampling: rejection sampling, likelihood weighting and Markov chain Monte Carlo (Gibbs sampling). Bayesian networks have been built for diagnosing livestock and plant diseases and for decision support in pest management.
A student must be able to state and use the bias-variance decomposition, set up k-fold, leave-one-out and blocked cross-validation, compute simple quantities by hand (a kNN prediction, a Gini index, a confusion-matrix metric, a ridge coefficient in the orthonormal case, a CNN parameter count or output size, a k-means update, a Bayes-rule posterior, an A* step), and explain when each method is appropriate. In a thesis the student builds and honestly validates a model on agricultural data and explains it with importance measures or SHAP values.
Five years into work at an agritech company, a research institute or an analytics unit of a ministry, the practitioner builds yield, price, disease and image models end to end: assembling data, choosing baselines, tuning with nested and blocked cross-validation, guarding against leakage, explaining models to agronomists, and monitoring performance after deployment when seasons, sensors or farming practices change.
A senior data-science or statistics head with 25 years of experience decides where machine learning adds value over simpler methods, sets validation and documentation standards, manages data and labelling pipelines, weighs the cost of errors in advisories that farmers act on, and ensures fairness and accountability when models allocate subsidies, credit or insurance payouts.
A professor or research leader develops methods for agricultural learning problems: spatio-temporal and hierarchical models combined with machine learning, uncertainty quantification for forecasts, domain adaptation from laboratory to field images, causal machine learning for programme evaluation, and hybrid process-based and data-driven models. The research leader supervises theses, reviews papers whose accuracy claims rest on random splits of correlated data, and sets standards for reproducible, honestly validated agricultural AI.
This manual does not cover reinforcement learning, recurrent and transformer architectures in depth, Gaussian processes, Bayesian neural networks, causal machine learning, or the engineering of deployment (data pipelines, monitoring, model updating). Software libraries are not named or taught. Default hyperparameter values quoted (such as m about sqrt(p) for random forests) are common conventions, not rules, and differ between implementations.
Original practice questions written to the syllabus and pattern of the exams named in each question's examTags. They are not past papers and do not reproduce any board's questions. Every numerical answer was computed with a script; the values needed are stated in the question.
These notes are free to read here, with no account.
What the Laisha app adds:
Free for ten days, then a subscription. We would rather write that here than have you find it later.